[Gdal-dev] ECW compression - Extremely slow Linux performance

[Andrew Larcombe]

Andrew Larcombe wrote:
> Frank Warmerdam wrote:
>> Andrew Larcombe wrote:
>>> Frank - according to top the CPU usage varies but never gets above 
>>> 20%. Does this shed any more light on the issue?
>>
>> Andrew,
>>
>> This indicates things are stalling somewhere.  My guess then is that it
>> is a thread switching issue of some sort.  How confident are you that
>> you are using the latest and greatest ECW SDK?
>>
>> It could also be some sort of IO problem.  I think the encoder keeps
>> temp files in /tmp.  I trust that is a fast local disk too?
>>
>> Beyond that it gets pretty involved to try and diagnose.   I just hate
>> multithreaded libraries.

To get some more consistent results, I've tried comparing the 
performance of the latest FWTools on my laptop and the problematic 
server. Tests are done on local drives with /tmp also being local. Test 
is to convert a 4000x4000 uncompressed TIF to ECW with TARGET=90 set 
using gdal_convert found in the bin_safe directory:

Laptop:
Centrino 1.4GHz, 1Gig RAM
Mandriva 2006, glibc-2.3.5

16.44user
10.53system
0:59.02elapsed
45%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (0major+8878minor)pagefaults 0swaps

Server:
4x Dual-Core Xeon 3.66GHz, 8Gig RAM
Redhat Enterprise Linux 3, glibc-2.3.2

real    1m55.632s
user    0m5.360s
sys     0m1.380s

ie, the server takes ~2 minutes, the laptop ~1 minute which isn't what 
one would expect. Additionally, both are extremely slow compared to the 
FWTools shell on Windows XP laptop (1.6GHz, 512MB RAM) which processes 
it in ~6 seconds.

Compressing to JPEG using gdal_translate is a different matter - 
basically what you would expect. The laptop takes ~4 seconds, the server 
1.4 seconds.

Regards,

Andrew


-- 
Andrew Larcombe
Freelance Geospatial, Database & Web Programming

web: http://www.andrewlarcombe.co.uk
email: andrew at andrewlarcombe.co.uk
mob: +44 (7760) 258623
icq: 306690163