[Gdal-dev] ECW compression - Extremely slow Linux performance

[Andrew Larcombe]

Frank Warmerdam wrote:
> Andrew Larcombe wrote:
>> Frank - according to top the CPU usage varies but never gets above 
>> 20%. Does this shed any more light on the issue?
> 
> Andrew,
> 
> This indicates things are stalling somewhere.  My guess then is that it
> is a thread switching issue of some sort.  How confident are you that
> you are using the latest and greatest ECW SDK?
> 
> It could also be some sort of IO problem.  I think the encoder keeps
> temp files in /tmp.  I trust that is a fast local disk too?
> 
> Beyond that it gets pretty involved to try and diagnose.   I just hate
> multithreaded libraries.

Frank,

Yes /tmp is a fast local disk too. I'm using the latest source ECW SDK - 
3.3RC2. I note there is a 3.3 final release, but these are Windows binaries.

strace'ing gdal_translate gives this:

% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
  35.14    0.357212          14     26030           write
  24.54    0.249446           9     26948           sched_yield
  18.41    0.187088           7     25948           gettimeofday
   9.42    0.095770          17      5505           nanosleep
   7.40    0.075248           6     12238           _llseek
   1.68    0.017050          11      1576         1 read
   1.64    0.016643          65       257           munmap
   0.46    0.004717          15       317           mprotect
   0.32    0.003283         657         5           unlink
   0.29    0.002901          24       119           brk
   0.23    0.002337           9       264           mmap2
   0.13    0.001281          16        81        25 open
   0.09    0.000914          14        65           old_mmap
   0.06    0.000572          10        56           close
   0.06    0.000566           9        64           fstat64
   0.05    0.000463          10        46        21 stat64
   0.03    0.000289          58         5         5 lstat64
   0.02    0.000213          11        19         3 futex
   0.02    0.000172         172         1           execve
   0.01    0.000084          17         5           clone
   0.00    0.000033          17         2           tgkill
   0.00    0.000027           9         3           sched_getparam
   0.00    0.000026           9         3           sched_getscheduler
   0.00    0.000023          12         2           getrlimit
   0.00    0.000014           3         5           getpid
   0.00    0.000014          14         1           set_thread_area
   0.00    0.000013          13         1           rt_sigaction
   0.00    0.000011          11         1           rt_sigprocmask
   0.00    0.000011          11         1           set_tid_address
   0.00    0.000004           4         1           uname
------ ----------- ----------- --------- --------- ----------------
100.00    1.016425                 99569        55 total

Do you have any more thoughts on this, or where the problem could lie? 
It does seem to be making an awfully large number of gettimeofday calls, 
although maybe this is a red herring?

Regards,

Andrew

-- 
Andrew Larcombe
andrew at andrewlarcombe.co.uk