<div dir="ltr"><div class="gmail_extra"><div>Hi Even,</div><div>To hit the 10GB limit I'm using (Windows Server 2008):</div><div><br></div><div>gdal_translate -of GTiff -co TILED=YES -co BIGTIFF=YES -co COMPRESS=JPEG -co JPEG_QUALITY=60 -co BLOCKXSIZE=512 -co BLOCKYSIZE=512 -co PHOTOMETRIC=YCBCR 2000.vrt 2000.tif<br>
</div><div><br></div><div>I tried adding: --config GDAL_CACHEMAX 27000000000 but then it'd just crash with out-of-memory (it wouldn't actually stop at 27GB; so I guess that works differently again).</div><div><br>
</div><div>====</div><div><br></div><div>I was also using this (which is a simple tif to tif) and it too used 10GB before stopping itself (this was Windows 7):</div><div>call gdal_merge -o output.tif -of GTiff -co TILED=YES -co BIGTIFF=YES -co COMPRESS=JPEG -co JPEG_QUALITY=80 -co BLOCKXSIZE=512 -co BLOCKYSIZE=512 --optfile tiff_list.txt -co PHOTOMETRIC=YCBCR<br>
</div><div><br></div><div>It may just have been co-incident.</div><div><br></div><div>I think I'll just let it do it automatically and hope for the best.<br><br></div><div>Jonathan</div>
<br><br><div class="gmail_quote">On 2 December 2013 17:57, Even Rouault <span dir="ltr"><<a href="mailto:even.rouault@mines-paris.org" target="_blank">even.rouault@mines-paris.org</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
Le lundi 02 décembre 2013 18:49:42, Jonathan Moules a écrit :<br>
<div class="im">> Hi Even,<br>
> A further thought while we're on the issue of limits, it appears that gdal<br>
> (both gdal_translate and gdalwarp) have an internal limit of 10GB when<br>
> they're auto-allocating RAM.<br>
<br>
</div>I'm not aware of such a limitation in the code. It might be that your OS<br>
cannot allocate that much memory in one single malloc() if it is the<br>
allocation pattern involved here, but I'm just guessing. What commands did you<br>
exactly tried to trigger this ?<br>
<div class="im"><br>
><br>
> Assuming this is the case and it's not just coincidence, it might it be<br>
> worth updating them to allow more use if appropriate (it seems to be<br>
> hitting this for a large-ecw to geotiff convesion)?<br>
<br>
</div>The ECW SDK itself has its own RAM management. See<br>
<a href="http://www.gdal.org/frmt_ecw.html" target="_blank">http://www.gdal.org/frmt_ecw.html</a><br>
<div class=""><div class="h5"><br>
> For instance my work<br>
> desktop has 16GB and the server I'm also doing some processing on has 36GB.<br>
><br>
> Cheers,<br>
> Jonathan<br>
><br>
> (also I failed to reply-all to my previous reply; so included below)<br>
><br>
><br>
> On 2 December 2013 17:25, Jonathan Moules <<br>
><br>
> <a href="mailto:jonathanmoules@warwickshire.gov.uk">jonathanmoules@warwickshire.gov.uk</a>> wrote:<br>
> > Hi Even,<br>
> > Thanks for the detailed reply. It seems in the end that I can use a<br>
> > combination of gdalbuildvrt and gdal_translate to get the same results<br>
> > without having to bother with manual memory management. The output is the<br>
> > same size and the quality is identical.<br>
> ><br>
> > From a user perspective, I would suggest removing this behaviour (though<br>
> > documenting would work too). You could add a gdal warning if they set it<br>
> > above 10GB with a similar caveat to the "Note" in your email.<br>
> > Of course, then in another 7 years you'll be asked why the warning is<br>
> > triggering at the "low" 10GB!<br>
> ><br>
> > Thanks again,<br>
> > Jonathan<br>
> ><br>
> > On 2 December 2013 16:39, Even Rouault <even.rouault@mines-<br>
<a href="http://paris.org" target="_blank">paris.org</a>>wrote:<br>
> >> Le lundi 02 décembre 2013 17:17:11, Jonathan Moules a écrit :<br>
> >> > Hi List,<br>
> >> > I've just tried the following:<br>
> >> ><br>
> >> > call gdalwarp -of GTiff *-wm 13000* -co TILED=YES -co BIGTIFF=YES -co<br>
> >> > COMPRESS=JPEG -co JPEG_QUALITY=80 -co BLOCKXSIZE=512 -co<br>
> >> > BLOCKYSIZE=512<br>
> >><br>
> >> -co<br>
> >><br>
> >> > PHOTOMETRIC=YCBCR SP1937.tif SP1938.tif SP1939.tif 1.tif<br>
> >> ><br>
> >> > The documentation says:<br>
> >> > > -wm memory_in_mb:<br>
> >> > > Set the amount of memory (in megabytes) that the warp API is allowed<br>
> >><br>
> >> to<br>
> >><br>
> >> > > use for caching.<br>
> >> ><br>
> >> > So by my reading I'm allocating a bit less than 13GB (of my<br>
> >> > 16,730,672kB RAM) to gdalwarp.<br>
> >> ><br>
> >> > Imagine my surprise when I get this error:<br>
> >> > ERROR 5: GDALWarpOptions.Validate()<br>
> >> ><br>
> >> > dfWarpMemoryLimit=13000 is unreasonably small.<br>
> >> ><br>
> >> > After more testing, anything with 4 digits works fine (including<br>
> >> > 9999),<br>
> >><br>
> >> but<br>
> >><br>
> >> > anything with five digits (i.e. 10,000) gives me that error.<br>
> >> ><br>
> >> > Is this a bug?<br>
> >><br>
> >> Rather a undocumented attempt to correct wrong user supplied parameter.<br>
> >><br>
> >> In gdalwarp utility, if the value of -wm is < 10000, then it is<br>
> >> considered as<br>
> >> megabytes ( and multiplied by 1024*1024 for GDAL internals), otherwise<br>
> >> it is<br>
> >> considered as bytes, and passed unmodified to GDAL internals. (the last<br>
> >> change<br>
> >> to that logic was in <a href="http://trac.osgeo.org/gdal/changeset/10817" target="_blank">http://trac.osgeo.org/gdal/changeset/10817</a> , 7<br>
> >> years ago,<br>
> >> where the threshold was modified from 4000 to 10000, so at that time 10<br>
> >> GB was<br>
> >> really enormous !)<br>
> >> In gdal warper code, there is a test that checks if the memory limit<br>
> >> variable<br>
> >> (that must be in bytes now) is at least 100000, and bail out if it is<br>
> >> not the<br>
> >> case.<br>
> >> So currently you could specify the size as bytes : "-wm 16730672000".<br>
> >><br>
> >> I'm wondering if we should not remove this GDAL-isense logic in gdalwarp<br>
> >> utility and just implement documented behaviour... Or maybe document the<br>
> >> current behaviour.<br>
> >><br>
> >> Note: unless your warping process implies considerable image shape<br>
> >> distortion<br>
> >> (which is unlikely to be the case here since I don't see any -s_srs or<br>
> >> -t_srs<br>
> >> flag), you don't need such a huge value. It could decrease performance<br>
> >> actually. The only case where it might help is if your source images are<br>
> >> not<br>
> >> tiled.<br>
> >><br>
> >> > (GDAL 1.10.1)<br>
> >> ><br>
> >> > Cheers,<br>
> >> > Jonathan<br>
> >><br>
> >> --<br>
> >> Geospatial professional services<br>
> >> <a href="http://even.rouault.free.fr/services.html" target="_blank">http://even.rouault.free.fr/services.html</a><br>
<br>
--<br>
Geospatial professional services<br>
<a href="http://even.rouault.free.fr/services.html" target="_blank">http://even.rouault.free.fr/services.html</a><br>
</div></div></blockquote></div><br></div></div>
<br>
<span style="color:rgb(34,34,34);font-family:arial,sans-serif;background-color:rgb(255,255,255)">This transmission is intended for the named addressee(s) only and may contain sensitive or protectively marked material up to RESTRICTED and should be handled accordingly. Unless you are the named addressee (or authorised to receive it for the addressee) you may not copy or use it, or disclose it to anyone else. If you have received this transmission in error please notify the sender immediately. All email traffic sent to or from us, including without limitation all GCSX traffic, may be subject to recording and/or monitoring in accordance with relevant legislation.</span>