[gdal-dev] -wm (memory in mb with gdalwarp)

[Jonathan Moules]

Hi Even,
A further thought while we're on the issue of limits, it appears that gdal
(both gdal_translate and gdalwarp) have an internal limit of 10GB when
they're auto-allocating RAM.

Assuming this is the case and it's not just coincidence, it might it be
worth updating them to allow more use if appropriate (it seems to be
hitting this for a large-ecw to geotiff convesion)? For instance my work
desktop has 16GB and the server I'm also doing some processing on has 36GB.

Cheers,
Jonathan

(also I failed to reply-all to my previous reply; so included below)


On 2 December 2013 17:25, Jonathan Moules <
jonathanmoules at warwickshire.gov.uk> wrote:

> Hi Even,
> Thanks for the detailed reply. It seems in the end that I can use a
> combination of gdalbuildvrt and gdal_translate to get the same results
> without having to bother with manual memory management. The output is the
> same size and the quality is identical.
>
> From a user perspective, I would suggest removing this behaviour (though
> documenting would work too). You could add a gdal warning if they set it
> above 10GB with a similar caveat to the "Note" in your email.
> Of course, then in another 7 years you'll be asked why the warning is
> triggering at the "low" 10GB!
>
> Thanks again,
> Jonathan
>
>
> On 2 December 2013 16:39, Even Rouault <even.rouault at mines-paris.org>wrote:
>
>> Le lundi 02 décembre 2013 17:17:11, Jonathan Moules a écrit :
>> > Hi List,
>> > I've just tried the following:
>> >
>> > call gdalwarp -of GTiff *-wm 13000* -co TILED=YES -co BIGTIFF=YES -co
>> > COMPRESS=JPEG -co JPEG_QUALITY=80 -co BLOCKXSIZE=512 -co BLOCKYSIZE=512
>> -co
>> > PHOTOMETRIC=YCBCR SP1937.tif SP1938.tif SP1939.tif 1.tif
>> >
>> > The documentation says:
>> > > -wm memory_in_mb:
>> > > Set the amount of memory (in megabytes) that the warp API is allowed
>> to
>> > > use for caching.
>> >
>> > So by my reading I'm allocating a bit less than 13GB (of my 16,730,672kB
>> > RAM) to gdalwarp.
>> >
>> > Imagine my surprise when I get this error:
>> >  ERROR 5: GDALWarpOptions.Validate()
>> >   dfWarpMemoryLimit=13000 is unreasonably small.
>> >
>> > After more testing, anything with 4 digits works fine (including 9999),
>> but
>> > anything with five digits (i.e. 10,000) gives me that error.
>> >
>> > Is this a bug?
>>
>> Rather a undocumented attempt to correct wrong user supplied parameter.
>>
>> In gdalwarp utility, if the value of -wm is < 10000, then it is
>> considered as
>> megabytes ( and multiplied by 1024*1024 for GDAL internals), otherwise it
>> is
>> considered as bytes, and passed unmodified to GDAL internals. (the last
>> change
>> to that logic was in http://trac.osgeo.org/gdal/changeset/10817 , 7
>> years ago,
>> where the threshold was modified from 4000 to 10000, so at that time 10
>> GB was
>> really enormous !)
>> In gdal warper code, there is a test that checks if the memory limit
>> variable
>> (that must be in bytes now) is at least 100000, and bail out if it is not
>> the
>> case.
>> So currently you could specify the size as bytes : "-wm 16730672000".
>>
>> I'm wondering if we should not remove this GDAL-isense logic in gdalwarp
>> utility and just implement documented behaviour... Or maybe document the
>> current behaviour.
>>
>> Note: unless your warping process implies considerable image shape
>> distortion
>> (which is unlikely to be the case here since I don't see any -s_srs or
>> -t_srs
>> flag), you don't need such a huge value. It could decrease performance
>> actually. The only case where it might help is if your source images are
>> not
>> tiled.
>>
>> >
>> >  (GDAL 1.10.1)
>> >
>> > Cheers,
>> > Jonathan
>>
>> --
>> Geospatial professional services
>> http://even.rouault.free.fr/services.html
>>
>
>

-- 
This transmission is intended for the named addressee(s) only and may 
contain sensitive or protectively marked material up to RESTRICTED and 
should be handled accordingly. Unless you are the named addressee (or 
authorised to receive it for the addressee) you may not copy or use it, or 
disclose it to anyone else. If you have received this transmission in error 
please notify the sender immediately. All email traffic sent to or from us, 
including without limitation all GCSX traffic, may be subject to recording 
and/or monitoring in accordance with relevant legislation.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20131202/c56da72b/attachment.html>