[gdal-dev] gdal_merge is very slow

[Greg Coats]

Since you have 4 GB RAM, I suggest overriding the 40 MB GDAL_CACHEMAX default, and increasing it
gdal_merge.py --config GDAL_CACHEMAX 1000 
The maximum possible GDAL_CACHEMAX is
gdal_merge.py --config GDAL_CACHEMAX 2147
Greg

On Jul 16, 2010, at 9:19 PM, Leith Bade wrote:

> Hi,
> 
> I am trying to use gdal_merge to mosaic a very large topo GeoTIFF set.
> 
> Uncompressed the data set is 60GB, but I keep it stored with DEFLATE compression which results in a dataset under 10GB.
> 
> Mosaicked the uncompressed file will be 125GB because of the large regions of nodata generated. Unfortunately this is too big to store on my HDD so I need to apply DEFLATE to it as well.
> 
> I am experimenting with only 1/2 the data set at the moment with this command:
> gdal_merge -co COMPRESS=DEFLATE -co ZLEVEL=9 -co BIGTIFF=YES -o NI-50.tif *-00.tif
> 
> On my AMD Phenom II X4 @ 3.2GHz, 64 bit Windows 7, 4GB DDR3 CAS-7 RAM (basically a decently specced PC gaming machine)
> it has been running for 16 hours and so far has only made it 60% (124 of 204 files)
> 
> The other issue is so far the file is 54GB so I will likely run out of disc space before it finished. This indicates that no DEFLATE compression is happening at all!
> 
> It currently maxing only 1 CPU core, so I assume it is trying to run DEFLATE then somehow failing to compress at all?
> 
> Looking at gdal_merge.py I think the major performance issue is the order in which it copies the data. Currently it copies an entire image at a time (1 colour channel at a time). Thus DEFLATE will not work due to the rather random write pattern to each scanline.
> 
> I think a much faster method would be to calculate the destination rectangles for each file into some sort of list. Aftter this generate the destination file 1 scanline at a time, calculating which source images intersect, and working left to right filling with solid color or copying scanlines from the source image.
> 
> This allows the DEFLATE to work far more effciently as the write pattern is horizontally linear.
> 
> What are your views/suggestions/etc.?
> 
> Thanks,
> Leith Bade
> leith at leithalweapon.geek.nz
> 
> _______________________________________________
> gdal-dev mailing list
> gdal-dev at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/gdal-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.osgeo.org/pipermail/gdal-dev/attachments/20100716/804edbac/attachment.html