[gdal-dev] -wm (memory in mb with gdalwarp)

[Even Rouault]

Le lundi 02 décembre 2013 17:17:11, Jonathan Moules a écrit :
> Hi List,
> I've just tried the following:
> 
> call gdalwarp -of GTiff *-wm 13000* -co TILED=YES -co BIGTIFF=YES -co
> COMPRESS=JPEG -co JPEG_QUALITY=80 -co BLOCKXSIZE=512 -co BLOCKYSIZE=512 -co
> PHOTOMETRIC=YCBCR SP1937.tif SP1938.tif SP1939.tif 1.tif
> 
> The documentation says:
> > -wm memory_in_mb:
> > Set the amount of memory (in megabytes) that the warp API is allowed to
> > use for caching.
> 
> So by my reading I'm allocating a bit less than 13GB (of my 16,730,672kB
> RAM) to gdalwarp.
> 
> Imagine my surprise when I get this error:
>  ERROR 5: GDALWarpOptions.Validate()
>   dfWarpMemoryLimit=13000 is unreasonably small.
> 
> After more testing, anything with 4 digits works fine (including 9999), but
> anything with five digits (i.e. 10,000) gives me that error.
> 
> Is this a bug?

Rather a undocumented attempt to correct wrong user supplied parameter.

In gdalwarp utility, if the value of -wm is < 10000, then it is considered as 
megabytes ( and multiplied by 1024*1024 for GDAL internals), otherwise it is 
considered as bytes, and passed unmodified to GDAL internals. (the last change 
to that logic was in http://trac.osgeo.org/gdal/changeset/10817 , 7 years ago, 
where the threshold was modified from 4000 to 10000, so at that time 10 GB was 
really enormous !)
In gdal warper code, there is a test that checks if the memory limit variable 
(that must be in bytes now) is at least 100000, and bail out if it is not the 
case.
So currently you could specify the size as bytes : "-wm 16730672000".

I'm wondering if we should not remove this GDAL-isense logic in gdalwarp 
utility and just implement documented behaviour... Or maybe document the 
current behaviour.

Note: unless your warping process implies considerable image shape distortion 
(which is unlikely to be the case here since I don't see any -s_srs or -t_srs 
flag), you don't need such a huge value. It could decrease performance 
actually. The only case where it might help is if your source images are not 
tiled.

> 
>  (GDAL 1.10.1)
> 
> Cheers,
> Jonathan

-- 
Geospatial professional services
http://even.rouault.free.fr/services.html