<div dir="ltr"><div>Hi Even,</div>A further thought while we're on the issue of limits, it appears that gdal (both gdal_translate and gdalwarp) have an internal limit of 10GB when they're auto-allocating RAM.<br><div class="gmail_extra">
<br></div><div class="gmail_extra">
Assuming this is the case and it's not just coincidence, it might it be worth updating them to allow more use if appropriate (it seems to be hitting this for a large-ecw to geotiff convesion)? For instance my work desktop has 16GB and the server I'm also doing some processing on has 36GB.</div>
<div class="gmail_extra"><br></div><div class="gmail_extra">Cheers,</div><div class="gmail_extra">Jonathan</div><div class="gmail_extra"><br></div><div class="gmail_extra">(also I failed to reply-all to my previous reply; so included below)</div>
<div class="gmail_extra"><br><br><div class="gmail_quote">On 2 December 2013 17:25, Jonathan Moules <span dir="ltr"><<a href="mailto:jonathanmoules@warwickshire.gov.uk" target="_blank">jonathanmoules@warwickshire.gov.uk</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hi Even,<div>Thanks for the detailed reply. It seems in the end that I can use a combination of gdalbuildvrt and gdal_translate to get the same results without having to bother with manual memory management. The output is the same size and the quality is identical.<br>
<div class="gmail_extra"><div><br></div>
>From a user perspective, I would suggest removing this behaviour (though documenting would work too). You could add a gdal warning if they set it above 10GB with a similar caveat to the "Note" in your email.</div>
<div class="gmail_extra">Of course, then in another 7 years you'll be asked why the warning is triggering at the "low" 10GB!</div><div class="gmail_extra"><br></div><div class="gmail_extra">Thanks again,</div>
<div class="gmail_extra">Jonathan<div><div class="h5"><br><br><div class="gmail_quote">On 2 December 2013 16:39, Even Rouault <span dir="ltr"><<a href="mailto:even.rouault@mines-paris.org" target="_blank">even.rouault@mines-paris.org</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">Le lundi 02 décembre 2013 17:17:11, Jonathan Moules a écrit :<br>
<div>> Hi List,<br>
> I've just tried the following:<br>
><br>
</div>> call gdalwarp -of GTiff *-wm 13000* -co TILED=YES -co BIGTIFF=YES -co<br>
<div>> COMPRESS=JPEG -co JPEG_QUALITY=80 -co BLOCKXSIZE=512 -co BLOCKYSIZE=512 -co<br>
> PHOTOMETRIC=YCBCR SP1937.tif SP1938.tif SP1939.tif 1.tif<br>
><br>
> The documentation says:<br>
> > -wm memory_in_mb:<br>
> > Set the amount of memory (in megabytes) that the warp API is allowed to<br>
> > use for caching.<br>
><br>
> So by my reading I'm allocating a bit less than 13GB (of my 16,730,672kB<br>
> RAM) to gdalwarp.<br>
><br>
> Imagine my surprise when I get this error:<br>
> ERROR 5: GDALWarpOptions.Validate()<br>
> dfWarpMemoryLimit=13000 is unreasonably small.<br>
><br>
> After more testing, anything with 4 digits works fine (including 9999), but<br>
> anything with five digits (i.e. 10,000) gives me that error.<br>
><br>
> Is this a bug?<br>
<br>
</div>Rather a undocumented attempt to correct wrong user supplied parameter.<br>
<br>
In gdalwarp utility, if the value of -wm is < 10000, then it is considered as<br>
megabytes ( and multiplied by 1024*1024 for GDAL internals), otherwise it is<br>
considered as bytes, and passed unmodified to GDAL internals. (the last change<br>
to that logic was in <a href="http://trac.osgeo.org/gdal/changeset/10817" target="_blank">http://trac.osgeo.org/gdal/changeset/10817</a> , 7 years ago,<br>
where the threshold was modified from 4000 to 10000, so at that time 10 GB was<br>
really enormous !)<br>
In gdal warper code, there is a test that checks if the memory limit variable<br>
(that must be in bytes now) is at least 100000, and bail out if it is not the<br>
case.<br>
So currently you could specify the size as bytes : "-wm 16730672000".<br>
<br>
I'm wondering if we should not remove this GDAL-isense logic in gdalwarp<br>
utility and just implement documented behaviour... Or maybe document the<br>
current behaviour.<br>
<br>
Note: unless your warping process implies considerable image shape distortion<br>
(which is unlikely to be the case here since I don't see any -s_srs or -t_srs<br>
flag), you don't need such a huge value. It could decrease performance<br>
actually. The only case where it might help is if your source images are not<br>
tiled.<br>
<br>
><br>
> (GDAL 1.10.1)<br>
><br>
> Cheers,<br>
> Jonathan<br>
<span><font color="#888888"><br>
--<br>
Geospatial professional services<br>
<a href="http://even.rouault.free.fr/services.html" target="_blank">http://even.rouault.free.fr/services.html</a><br>
</font></span></blockquote></div><br></div></div></div></div></div>
</blockquote></div><br></div></div>
<br>
<span style="color:rgb(34,34,34);font-family:arial,sans-serif;background-color:rgb(255,255,255)">This transmission is intended for the named addressee(s) only and may contain sensitive or protectively marked material up to RESTRICTED and should be handled accordingly. Unless you are the named addressee (or authorised to receive it for the addressee) you may not copy or use it, or disclose it to anyone else. If you have received this transmission in error please notify the sender immediately. All email traffic sent to or from us, including without limitation all GCSX traffic, may be subject to recording and/or monitoring in accordance with relevant legislation.</span>