[gdal-dev] gdal_merge is very slow

[Leith Bade]

OK I will try the tiling.

The aim of this merge is to generate an input for creating a compressed JPEG
2000 image (either using GDAL, Kakadu tools or the ER Image Compressor).

In the meantime I have installed the 30-day demo of ER Mapper. This program
was able to generate a mosaic in only a few seconds, and is currently
merging the image and compressing it in one step, estimated time is only
30mins.

This is a LOT faster than GDAL, but unfortunately I can't afford the full
licence (this is a small project that I am trying to do for free). If I
can't get GDAL to work nicely I might be forced to use the 30 day demo and
just reinstall it when it expires...

One thing I note is that it doesn't use more than 1 CPU to compress, while
Kakadu will happily use all 4 CPUs and all my memory to get it done in very
short time.

I might end up using ER Mapper to produce a large GTiff for kdu_compress to
work on.

Thanks,
Leith Bade
leith at leithalweapon.geek.nz



On 17 July 2010 14:22, Greg Coats <gregcoats at mac.com> wrote:

> I do not see that you specified that the output TIFF image be tiled
> -co TILED=YES -co BLOCKXSIZE=512 -co BLOCKYSIZE=512
> gdal_merge.py supports the -v input option that reports progress as a % for
> every source image merged into the destination image.
> It is better to do the initial gdal_merge.py uncompressed, and then follow
> that with a gdal_translate to created a compressed file.
> Greg
>
> On Jul 16, 2010, at 9:19 PM, Leith Bade wrote:
>
> Hi,
>
> I am trying to use gdal_merge to mosaic a very large topo GeoTIFF set.
>
> Uncompressed the data set is 60GB, but I keep it stored with DEFLATE
> compression which results in a dataset under 10GB.
>
> Mosaicked the uncompressed file will be 125GB because of the large regions
> of nodata generated. Unfortunately this is too big to store on my HDD so I
> need to apply DEFLATE to it as well.
>
> I am experimenting with only 1/2 the data set at the moment with this
> command:
> gdal_merge -co COMPRESS=DEFLATE -co ZLEVEL=9 -co BIGTIFF=YES -o NI-50.tif
> *-00.tif
>
> On my AMD Phenom II X4 @ 3.2GHz, 64 bit Windows 7, 4GB DDR3 CAS-7 RAM
> (basically a decently specced PC gaming machine)
> it has been running for 16 hours and so far has only made it 60% (124 of
> 204 files)
>
> The other issue is so far the file is 54GB so I will likely run out of disc
> space before it finished. This indicates that no DEFLATE compression is
> happening at all!
>
> It currently maxing only 1 CPU core, so I assume it is trying to run
> DEFLATE then somehow failing to compress at all?
>
> Looking at gdal_merge.py I think the major performance issue is the order
> in which it copies the data. Currently it copies an entire image at a time
> (1 colour channel at a time). Thus DEFLATE will not work due to the rather
> random write pattern to each scanline.
>
> I think a much faster method would be to calculate the destination
> rectangles for each file into some sort of list. Aftter this generate the
> destination file 1 scanline at a time, calculating which source images
> intersect, and working left to right filling with solid color or copying
> scanlines from the source image.
>
> This allows the DEFLATE to work far more effciently as the write pattern is
> horizontally linear.
>
> What are your views/suggestions/etc.?
>
> Thanks,
> Leith Bade
> leith at leithalweapon.geek.nz
>
>  _______________________________________________
> gdal-dev mailing list
> gdal-dev at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/gdal-dev
>
>
>
> _______________________________________________
> gdal-dev mailing list
> gdal-dev at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/gdal-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.osgeo.org/pipermail/gdal-dev/attachments/20100717/be4902bb/attachment-0001.html