[gdal-dev] gdal_merge is very slow

[Even Rouault]

Leith,

For the record, I've just made changes in the development version that allow 
specifying more than 2 GB (the effective value might be 2GB or 3GB on 32 bit 
operating systems). See http://trac.osgeo.org/gdal/ticket/3689

And the current limit is 2047, not 2147. (2147 * 1024 * 1024 is a negative 
value, so the effect should be to have a 0-byte cache... This was a typo in the 
doc at http://trac.osgeo.org/gdal/wiki/ConfigOptions)

As you've noticed gdal_merge is not able to merge big images without requiring 
big amount of RAM. You can try gdalwarp instead.

Best regards,

Even

Le dimanche 18 juillet 2010 06:44:20, Leith Bade a écrit :
> That seemed to do the trick. It finished overnight, a lot faster than
> before.
> 
> Final file was 83GB. (Only 1/2 the dataset was used)
> 
> Is there a reason GDAL_CACHEMAX 2147 is the maximum? Under 64-bit a 32-bit
> process has 4GB of virtual memory rather than 2GB under 32-bit (or 3GB with
> the /3GB Windows boot switch).
> 
> I was successfully able to use kdu_compress on the result as well so now I
> only need 4GB to store the file! (Although kdu_compress also has a memory
> limit at 2GB, but you can flush the codestream after every tile to fix
> that).
> 
> Thanks,
> Leith Bade
> leith at leithalweapon.geek.nz
> 
> On 17 July 2010 15:29, Greg Coats <gregcoats at mac.com> wrote:
> > Since you have 4 GB RAM, I suggest overriding the 40 MB GDAL_CACHEMAX
> > default, and increasing it
> > gdal_merge.py --config GDAL_CACHEMAX 1000
> > The maximum possible GDAL_CACHEMAX is
> > gdal_merge.py --config GDAL_CACHEMAX 2147
> > Greg
> > 
> > On Jul 16, 2010, at 9:19 PM, Leith Bade wrote:
> > 
> > Hi,
> > 
> > I am trying to use gdal_merge to mosaic a very large topo GeoTIFF set.
> > 
> > Uncompressed the data set is 60GB, but I keep it stored with DEFLATE
> > compression which results in a dataset under 10GB.
> > 
> > Mosaicked the uncompressed file will be 125GB because of the large
> > regions of nodata generated. Unfortunately this is too big to store on
> > my HDD so I need to apply DEFLATE to it as well.
> > 
> > I am experimenting with only 1/2 the data set at the moment with this
> > command:
> > gdal_merge -co COMPRESS=DEFLATE -co ZLEVEL=9 -co BIGTIFF=YES -o NI-50.tif
> > *-00.tif
> > 
> > On my AMD Phenom II X4 @ 3.2GHz, 64 bit Windows 7, 4GB DDR3 CAS-7 RAM
> > (basically a decently specced PC gaming machine)
> > it has been running for 16 hours and so far has only made it 60% (124 of
> > 204 files)
> > 
> > The other issue is so far the file is 54GB so I will likely run out of
> > disc space before it finished. This indicates that no DEFLATE
> > compression is happening at all!
> > 
> > It currently maxing only 1 CPU core, so I assume it is trying to run
> > DEFLATE then somehow failing to compress at all?
> > 
> > Looking at gdal_merge.py I think the major performance issue is the order
> > in which it copies the data. Currently it copies an entire image at a
> > time (1 colour channel at a time). Thus DEFLATE will not work due to the
> > rather random write pattern to each scanline.
> > 
> > I think a much faster method would be to calculate the destination
> > rectangles for each file into some sort of list. Aftter this generate the
> > destination file 1 scanline at a time, calculating which source images
> > intersect, and working left to right filling with solid color or copying
> > scanlines from the source image.
> > 
> > This allows the DEFLATE to work far more effciently as the write pattern
> > is horizontally linear.
> > 
> > What are your views/suggestions/etc.?
> > 
> > Thanks,
> > Leith Bade
> > leith at leithalweapon.geek.nz
> > 
> >  _______________________________________________
> > 
> > gdal-dev mailing list
> > gdal-dev at lists.osgeo.org
> > http://lists.osgeo.org/mailman/listinfo/gdal-dev
> > 
> > 
> > 
> > _______________________________________________
> > gdal-dev mailing list
> > gdal-dev at lists.osgeo.org
> > http://lists.osgeo.org/mailman/listinfo/gdal-dev