<html><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">
I support Frank's suggestion, and note that it is my understanding that when the user does not specify a value for --config GDAL_CACHEMAX, then the default of (only) 40 MB is used.<div><div><br></div><div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><font face="Lucida Grande" size="3" style="font: 12.0px Lucida Grande">In January 2009, working with GDAL version 1.6.0, I did some testing and documenting of the effect of </font><font face="Courier" size="3" style="font: 12.0px Courier">GDAL_CACHEMAX</font><font face="Lucida Grande" size="3" style="font: 12.0px Lucida Grande">. Using as input an uncompressed GeoTif 15,000 x 15,000 x 3 = 675 MB, organized into 1024 x 1024 tiles, I found that using gdal_translate to convert this GeoTif to a GeoJP2 with the default </font><font face="Courier" size="3" style="font: 12.0px Courier">--config GDAL_CACHEMAX 40</font><font face="Lucida Grande" size="3" style="font: 12.0px Lucida Grande"> took 10473 sec (= 175 minutes), but after increasing </font><font face="Courier" size="3" style="font: 12.0px Courier">GDAL_CACHEMAX</font><font face="Lucida Grande" size="3" style="font: 12.0px Lucida Grande"> by only 12.5% to </font><font face="Courier" size="3" style="font: 12.0px Courier">--config GDAL_CACHEMAX 45</font><font face="Lucida Grande" size="3" style="font: 12.0px Lucida Grande"> the processing took only 166 sec (= 2.7 minutes), which is 63 times faster!</font></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><font face="Lucida Grande" size="3" style="font: 12.0px Lucida Grande"></font><font face="Lucida Grande" size="3" style="font: 12.0px Lucida Grande">Greg</font></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 12px/normal 'Lucida Grande'; min-height: 15px; "><br></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><font face="Courier" size="3" style="font: 12.0px Courier">GDAL 1.6.0, released 2008/12/04</font></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><font face="Courier" size="3" style="font: 12.0px Courier">Input GeoTif 15,000 x 15,000 x 3 = 675 MB</font></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><font face="Courier" size="3" style="font: 12.0px Courier">gdal_translate -of JP2KAK -co QUALITY=10 --config GDAL_CACHEMAX</font></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><font face="Courier" size="3" style="font: 12.0px Courier">0040 10473.14 sec</font></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><font face="Courier" size="3" style="font: 12.0px Courier">0041 10326.75 sec</font></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><font face="Courier" size="3" style="font: 12.0px Courier">0042 10492.85 sec</font></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><font face="Courier" size="3" style="font: 12.0px Courier">0043 7081.57 sec</font></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><font face="Courier" size="3" style="font: 12.0px Courier">0044 3503.33 sec</font></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><font face="Courier" size="3" style="font: 12.0px Courier">0045 166.23 sec</font></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><font face="Courier" size="3" style="font: 12.0px Courier">0050 166.31 sec</font></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><font face="Courier" size="3" style="font: 12.0px Courier">0055 165.89 sec</font></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><font face="Courier" size="3" style="font: 12.0px Courier">0060 165.78 sec</font></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><font face="Courier" size="3" style="font: 12.0px Courier">0080 165.48 sec</font></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><font face="Courier" size="3" style="font: 12.0px Courier">0100 166.53 sec</font></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><font face="Courier" size="3" style="font: 12.0px Courier">0125 165.59 sec</font></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><font face="Courier" size="3" style="font: 12.0px Courier">0150 165.72 sec</font></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><font face="Courier" size="3" style="font: 12.0px Courier">0200 165.65 sec</font></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><font class="Apple-style-span" face="Courier"><br></font></div></div><div><div><div>On Sep 16, 2009, at 9:42 AM, Frank Warmerdam wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><p style="margin: 0.0px 0.0px 0.0px 0.0px"><font face="Lucida Grande" size="3" style="font: 12.0px Lucida Grande">Hermann Peifer wrote:</font></p> <blockquote type="cite"><p style="margin: 0.0px 0.0px 0.0px 10.0px"><font face="Lucida Grande" size="3" style="font: 12.0px Lucida Grande">Hi,</font></p> <p style="margin: 0.0px 0.0px 0.0px 10.0px"><font face="Lucida Grande" size="3" style="font: 12.0px Lucida Grande">Are there any hints from the experts about how to improve gdal_rasterize perfomance?</font></p> <p style="margin: 0.0px 0.0px 0.0px 10.0px"><font face="Lucida Grande" size="3" style="font: 12.0px Lucida Grande">My specific case is to rasterize a 1+GB shapefile into a 26800 x 23200 GeoTiff. Are there some memory settings or GeoTiff creation options that could speed up the rasterization?</font></p> </blockquote><p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Lucida Grande; min-height: 15.0px"><br></p> <p style="margin: 0.0px 0.0px 0.0px 0.0px"><font face="Lucida Grande" size="3" style="font: 12.0px Lucida Grande">Hermann,</font></p> <p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Lucida Grande; min-height: 15.0px"><br></p> <p style="margin: 0.0px 0.0px 0.0px 0.0px"><font face="Lucida Grande" size="3" style="font: 12.0px Lucida Grande">This sounds like a heap of processing that needs to be done.</font></p> <p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Lucida Grande; min-height: 15.0px"><br></p> <p style="margin: 0.0px 0.0px 0.0px 0.0px"><font face="Lucida Grande" size="3" style="font: 12.0px Lucida Grande">You can increase the chunk size on which gdal_rasterize operates by</font></p> <p style="margin: 0.0px 0.0px 0.0px 0.0px"><font face="Lucida Grande" size="3" style="font: 12.0px Lucida Grande">increasing the GDAL_CACHEMAX config option.<span class="Apple-converted-space"> </span>I would suggest sizing</font></p> <p style="margin: 0.0px 0.0px 0.0px 0.0px"><font face="Lucida Grande" size="3" style="font: 12.0px Lucida Grande">it to be roughly 1/3 of your machine RAM.</font></p> <p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Lucida Grande; min-height: 15.0px"><br></p> <p style="margin: 0.0px 0.0px 0.0px 0.0px"><font face="Lucida Grande" size="3" style="font: 12.0px Lucida Grande">eg.</font></p> <p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Lucida Grande; min-height: 15.0px"><br></p> <p style="margin: 0.0px 0.0px 0.0px 0.0px"><font face="Lucida Grande" size="3" style="font: 12.0px Lucida Grande">gdal_rasterize --config GDAL_CACHEMAX 600 ...</font></p> <p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Lucida Grande; min-height: 15.0px"><br></p> <p style="margin: 0.0px 0.0px 0.0px 0.0px"><font face="Lucida Grande" size="3" style="font: 12.0px Lucida Grande">would be appropriate on a machine with 2GB of RAM.</font></p> <p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Lucida Grande; min-height: 15.0px"><br></p> <p style="margin: 0.0px 0.0px 0.0px 0.0px"><font face="Lucida Grande" size="3" style="font: 12.0px Lucida Grande">Beyond that I can't think of much that might help.<span class="Apple-converted-space"> </span>The above basically</font></p> <p style="margin: 0.0px 0.0px 0.0px 0.0px"><font face="Lucida Grande" size="3" style="font: 12.0px Lucida Grande">reduces the number of passes needed through the vector data.<span class="Apple-converted-space"> </span>It is all</font></p> <p style="margin: 0.0px 0.0px 0.0px 0.0px"><font face="Lucida Grande" size="3" style="font: 12.0px Lucida Grande">processed once for each chunk of the raster that is processed.</font></p> <p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Lucida Grande; min-height: 15.0px"><br></p> <p style="margin: 0.0px 0.0px 0.0px 0.0px"><font face="Lucida Grande" size="3" style="font: 12.0px Lucida Grande">Best regards,</font></p> </blockquote></div><br></div></div></body></html>