[gdal-dev] warp memory limit default too low and does not have a config option

[Even Rouault]

Le lundi 06 février 2012 02:41:40, Etienne Tourigny a écrit :
> Hi all,
> 
> The default memory limit for warp operations is 64MB, which is far too
> low using recent hardware and leads to inefficiencies in I/O when
> warping large images.
> 
> This can be changed using the -wm option of gdalwarp, or if using the
> API change the value of GDALWarpOptions::dfWarpMemoryLimit.
> More information here:
> http://trac.osgeo.org/gdal/wiki/UserDocs/GdalWarp#WillincreasingRAMincrease
> thespeedofgdalwarp
> 
> However, it would be better to do both of the following:
> 
> 1) Change the default value of 64 MB to something higher, perhaps
> depending on the machine's configuration (say 1/4 of RAM) - this
> possibility is mentioned in gdalwarpoperation.cpp  .

Finding good default values is quite difficult and somehow arbitrary. There's 
however something to be aware : in the trac entry you quote above, there's a 
mention of a use case where more RAM for dfWarpMemoryLimit will actually 
decrease performance. So if you intend increasing it substantially (which 
would be the case for 1/4 of RAM on modern configurations), that might make 
things significantly worse for that use case. That's definitely something to at 
least benchmark and lilely fix in the warping code. The use case if I remember 
well is "gdalwarp smallimage bigimage" (the real use case being generally 
"gdalwarp smallimage1 smallimage2 ... smallimageN bigmosaicofallsmallimages"). 
If you increase dfWarpMemoryLimit currently, we will currently read an area of 
bigimage much larger than needed to include the extent of smallimage, which 
cause excessive read operations before warping, and excessive write 
afterwards.

> 2) Add a config option so that the default can be set for all
> operations by the user (which overrides 1, but can be overridden for
> each operation)

Not sure to understand how option 2) if different from the -wm option and how 
it would be implemented ...

> 
> 
> Any comments?
> 
> regards,
> Etienne
> _______________________________________________
> gdal-dev mailing list
> gdal-dev at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/gdal-dev