[gdal-dev] GDAL_retile slowdown at next level

[christian.mueller at nvoe.at]

Hi, I am the developer of the gdal_retile.py

I think the problem here is based on the following issue.

To make a pyramid with tiles in the Dimension  width * hight based and  
already tiled image with the same tile width and height, I have to  
load the base tile and all 8 neighbors touching the tile. As you  
stated, your tiles are on a remote file server. This must result in  
heavy network traffic.

Open gdal_retily.py with an editor and search for "class  
DataSetCache". You see a line

self.cacheSize=8

which means the the cache is holding 8 tiles. Try to increase this  
number. A good number would be

(pixel width of your first pyramid / tile width ) * 3  + 8

I hope you get no memory problems






Quoting acangi <aca at ngi.be>:

>
> Hello,
>
> It looks like gdal_retile.py slows down when processing a new level of the
> pyramid.
>
> Source files : 8065 geotiffs orthophotos of 4000 x 4000 pixels, in total 400
> GB.
>
> Command : gdal_retile.py -targetDir pyramid/ -levels 12 -v -s_srs
> /mnt/webgisdata/geoserver/config/3812.prj -ps 512 512 -r bilinear -co
> "COMPRESS=JPEG" -co "INTERLEAVE=PIXEL" -tileIndex orthocol -tileIndexField
> location /opt/geodata/ortholinks/*.tif
>
> Level 0 was processed in 6 days (500000 files, 1 file is processed in 1
> second, total 64 GB).
> Level 1 starts processing : 1 file is processed in 5 minutes. As it'd take
> me a year to finish the level, I stopped it.
>
> Restarted with the same command and -pyramidOnly parameter
>
> Level 1 was processed in 6 days (125000 files, 1 file is processed in 4
> seconds, total 17 GB).
> Level 2 starts processing : 1 file is processed in 3 minutes. It would take
> 2 monthes to finish this level.
>
> I'll wait a little more to see the evolution and then stop the process
> again, and restart it based on the level 1 files.
>
> Any idea why it slows down like this ?
>
> The source files and the output are on a CIFS fileserver, the computer
> processing gdal_retile is a 4 CPU, 2.5 GB RAM. top says :
>
> top - 10:12:57 up 7 days, 23:37,  1 user,  load average: 1.00, 1.06, 1.09
> Tasks:   1 total,   0 running,   1 sleeping,   0 stopped,   0 zombie
> Cpu(s):  1.2%us,  0.0%sy,  0.0%ni, 98.2%id,  0.0%wa,  0.0%hi,  0.0%si,
> 0.7%st
> Mem:   2560220k total,  2506852k used,    53368k free,    21316k buffers
> Swap:  2097144k total,     5496k used,  2091648k free,   949224k cached
>
>   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
>  4913 ctiadmin  20   0  243m 163m 8064 D    6  6.5 283:36.06 python
>
> Any idea appreciated,
>
> Alain
> --
> View this message in context:   
> http://osgeo-org.1803224.n2.nabble.com/GDAL-retile-slowdown-at-next-level-tp5034870p5034870.html
> Sent from the GDAL - Dev mailing list archive at Nabble.com.
> _______________________________________________
> gdal-dev mailing list
> gdal-dev at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/gdal-dev
>



----------------------------------------------------------------
This message was sent using IMP, the Internet Messaging Program.