OK I will try the tiling.<div><br></div><div>The aim of this merge is to generate an input for creating a compressed JPEG 2000 image (either using GDAL, Kakadu tools or the ER Image Compressor).</div><div><br></div><div>In the meantime I have installed the 30-day demo of ER Mapper. This program was able to generate a mosaic in only a few seconds, and is currently merging the image and compressing it in one step, estimated time is only 30mins.</div>
<div><br></div><div>This is a LOT faster than GDAL, but unfortunately I can't afford the full licence (this is a small project that I am trying to do for free). If I can't get GDAL to work nicely I might be forced to use the 30 day demo and just reinstall it when it expires...</div>
<div><br></div><div>One thing I note is that it doesn't use more than 1 CPU to compress, while Kakadu will happily use all 4 CPUs and all my memory to get it done in very short time.</div><div><br></div><div>I might end up using ER Mapper to produce a large GTiff for kdu_compress to work on.</div>
<div><br clear="all">Thanks,<br>Leith Bade<br><a href="mailto:leith@leithalweapon.geek.nz">leith@leithalweapon.geek.nz</a><br><br>
<br><br><div class="gmail_quote">On 17 July 2010 14:22, Greg Coats <span dir="ltr"><<a href="mailto:gregcoats@mac.com">gregcoats@mac.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
<div style="word-wrap:break-word"><div>I do not see that you specified that the output TIFF image be tiled</div><div><div>-co TILED=YES -co BLOCKXSIZE=512 -co BLOCKYSIZE=512</div><div>gdal_merge.py supports the -v input option that reports progress as a % for every source image merged into the destination image. </div>
</div><div><div>It is better to do the initial gdal_merge.py uncompressed, and then follow that with a gdal_translate to created a compressed file.</div></div><div>Greg</div><br><div><div><div></div><div class="h5"><div>
On Jul 16, 2010, at 9:19 PM, Leith Bade wrote:</div>
<br></div></div><blockquote type="cite"><div><div></div><div class="h5">Hi,<div><br></div><div>I am trying to use gdal_merge to mosaic a very large topo GeoTIFF set.</div><div><br></div><div>Uncompressed the data set is 60GB, but I keep it stored with DEFLATE compression which results in a dataset under 10GB.</div>
<div><br></div><div>Mosaicked the uncompressed file will be 125GB because of the large regions of nodata generated. Unfortunately this is too big to store on my HDD so I need to apply DEFLATE to it as well.</div><div><br>
</div><div>I am experimenting with only 1/2 the data set at the moment with this command:</div><div>gdal_merge -co COMPRESS=DEFLATE -co ZLEVEL=9 -co BIGTIFF=YES -o NI-50.tif *-00.tif</div><div><br></div><div>On my AMD Phenom II X4 @ 3.2GHz, 64 bit Windows 7, 4GB DDR3 CAS-7 RAM (basically a decently specced PC gaming machine)</div>
<div>it has been running for 16 hours and so far has only made it 60% (124 of 204 files)</div><div><br></div><div>The other issue is so far the file is 54GB so I will likely run out of disc space before it finished. This indicates that no DEFLATE compression is happening at all!</div>
<div><br></div><div>It currently maxing only 1 CPU core, so I assume it is trying to run DEFLATE then somehow failing to compress at all?</div><div><br></div><div>Looking at gdal_merge.py I think the major performance issue is the order in which it copies the data. Currently it copies an entire image at a time (1 colour channel at a time). Thus DEFLATE will not work due to the rather random write pattern to each scanline.</div>
<div><br></div><div>I think a much faster method would be to calculate the destination rectangles for each file into some sort of list. Aftter this generate the destination file 1 scanline at a time, calculating which source images intersect, and working left to right filling with solid color or copying scanlines from the source image.</div>
<div><br></div><div>This allows the DEFLATE to work far more effciently as the write pattern is horizontally linear.</div><div><br></div><div>What are your views/suggestions/etc.?</div><div><br clear="all">Thanks,<br>Leith Bade<br>
<a href="mailto:leith@leithalweapon.geek.nz" target="_blank">leith@leithalweapon.geek.nz</a><br><br>
</div></div></div>
_______________________________________________<br>gdal-dev mailing list<br><a href="mailto:gdal-dev@lists.osgeo.org" target="_blank">gdal-dev@lists.osgeo.org</a><br><a href="http://lists.osgeo.org/mailman/listinfo/gdal-dev" target="_blank">http://lists.osgeo.org/mailman/listinfo/gdal-dev</a></blockquote>
</div><br></div>
<br>_______________________________________________<br>
gdal-dev mailing list<br>
<a href="mailto:gdal-dev@lists.osgeo.org">gdal-dev@lists.osgeo.org</a><br>
<a href="http://lists.osgeo.org/mailman/listinfo/gdal-dev" target="_blank">http://lists.osgeo.org/mailman/listinfo/gdal-dev</a><br></blockquote></div><br></div>