[gdal-dev] GDAL_retile slowdown at next level

[acangi]

Hello,

It looks like gdal_retile.py slows down when processing a new level of the
pyramid. 

Source files : 8065 geotiffs orthophotos of 4000 x 4000 pixels, in total 400
GB.

Command : gdal_retile.py -targetDir pyramid/ -levels 12 -v -s_srs
/mnt/webgisdata/geoserver/config/3812.prj -ps 512 512 -r bilinear -co
"COMPRESS=JPEG" -co "INTERLEAVE=PIXEL" -tileIndex orthocol -tileIndexField
location /opt/geodata/ortholinks/*.tif

Level 0 was processed in 6 days (500000 files, 1 file is processed in 1
second, total 64 GB).
Level 1 starts processing : 1 file is processed in 5 minutes. As it'd take
me a year to finish the level, I stopped it.

Restarted with the same command and -pyramidOnly parameter

Level 1 was processed in 6 days (125000 files, 1 file is processed in 4
seconds, total 17 GB).
Level 2 starts processing : 1 file is processed in 3 minutes. It would take
2 monthes to finish this level.

I'll wait a little more to see the evolution and then stop the process
again, and restart it based on the level 1 files.

Any idea why it slows down like this ?

The source files and the output are on a CIFS fileserver, the computer
processing gdal_retile is a 4 CPU, 2.5 GB RAM. top says : 

top - 10:12:57 up 7 days, 23:37,  1 user,  load average: 1.00, 1.06, 1.09
Tasks:   1 total,   0 running,   1 sleeping,   0 stopped,   0 zombie
Cpu(s):  1.2%us,  0.0%sy,  0.0%ni, 98.2%id,  0.0%wa,  0.0%hi,  0.0%si, 
0.7%st
Mem:   2560220k total,  2506852k used,    53368k free,    21316k buffers
Swap:  2097144k total,     5496k used,  2091648k free,   949224k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 4913 ctiadmin  20   0  243m 163m 8064 D    6  6.5 283:36.06 python

Any idea appreciated,

Alain
-- 
View this message in context: http://osgeo-org.1803224.n2.nabble.com/GDAL-retile-slowdown-at-next-level-tp5034870p5034870.html
Sent from the GDAL - Dev mailing list archive at Nabble.com.