[gdal-dev] -wm (memory in mb with gdalwarp)

Even Rouault even.rouault at mines-paris.org
Mon Dec 2 10:32:29 PST 2013 

Le lundi 02 décembre 2013 19:12:07, Jonathan Moules a écrit :
> Hi Even,
> To hit the 10GB limit I'm using (Windows Server 2008):
> 
> gdal_translate -of GTiff -co TILED=YES -co BIGTIFF=YES -co COMPRESS=JPEG
> -co JPEG_QUALITY=60 -co BLOCKXSIZE=512 -co BLOCKYSIZE=512 -co
> PHOTOMETRIC=YCBCR 2000.vrt 2000.tif
> 
> I tried adding:  --config GDAL_CACHEMAX 27000000000 but then it'd just
> crash with out-of-memory (it wouldn't actually stop at 27GB; so I guess
> that works differently again).

Crash (Windows dialog) or "clean" exit with a GDAL error message ?
What are the files (TIFF, ECW, ...) in the VRT ?
I don't see anything wrong, and have not OS and hardware specs to reproduce 
that.

> 
> ====
> 
> I was also using this (which is a simple tif to tif) and it too used 10GB
> before stopping itself (this was Windows 7):
> call gdal_merge -o output.tif -of GTiff -co TILED=YES -co BIGTIFF=YES -co
> COMPRESS=JPEG -co JPEG_QUALITY=80 -co BLOCKXSIZE=512 -co BLOCKYSIZE=512
> --optfile tiff_list.txt -co PHOTOMETRIC=YCBCR
> 
> It may just have been co-incident.
> 
> I think I'll just let it do it automatically and hope for the best.

Yes, that's perhaps the best. Increasing cache size is not always the best 
thing to do. You can probably push it to 1 or 2 GB. Above, it is dubious that 
it will help.

> 
> Jonathan
> 
> On 2 December 2013 17:57, Even Rouault <even.rouault at mines-paris.org> wrote:
> > Le lundi 02 décembre 2013 18:49:42, Jonathan Moules a écrit :
> > > Hi Even,
> > > A further thought while we're on the issue of limits, it appears that
> > 
> > gdal
> > 
> > > (both gdal_translate and gdalwarp) have an internal limit of 10GB when
> > > they're auto-allocating RAM.
> > 
> > I'm not aware of such a limitation in the code. It might be that your OS
> > cannot allocate that much memory in one single malloc() if it is the
> > allocation pattern involved here, but I'm just guessing. What commands
> > did you
> > exactly tried to trigger this ?
> > 
> > > Assuming this is the case and it's not just coincidence, it might it be
> > > worth updating them to allow more use if appropriate (it seems to be
> > > hitting this for a large-ecw to geotiff convesion)?
> > 
> > The ECW SDK itself has its own RAM management. See
> > http://www.gdal.org/frmt_ecw.html
> > 
> > > For instance my work
> > > desktop has 16GB and the server I'm also doing some processing on has
> > 
> > 36GB.
> > 
> > > Cheers,
> > > Jonathan
> > > 
> > > (also I failed to reply-all to my previous reply; so included below)
> > > 
> > > 
> > > On 2 December 2013 17:25, Jonathan Moules <
> > > 
> > > jonathanmoules at warwickshire.gov.uk> wrote:
> > > > Hi Even,
> > > > Thanks for the detailed reply. It seems in the end that I can use a
> > > > combination of gdalbuildvrt and gdal_translate to get the same
> > > > results without having to bother with manual memory management. The
> > > > output is
> > 
> > the
> > 
> > > > same size and the quality is identical.
> > > > 
> > > > From a user perspective, I would suggest removing this behaviour
> > 
> > (though
> > 
> > > > documenting would work too). You could add a gdal warning if they set
> > 
> > it
> > 
> > > > above 10GB with a similar caveat to the "Note" in your email.
> > > > Of course, then in another 7 years you'll be asked why the warning is
> > > > triggering at the "low" 10GB!
> > > > 
> > > > Thanks again,
> > > > Jonathan
> > > > 
> > > > On 2 December 2013 16:39, Even Rouault <even.rouault at mines-
> > 
> > paris.org>wrote:
> > > >> Le lundi 02 décembre 2013 17:17:11, Jonathan Moules a écrit :
> > > >> > Hi List,
> > > >> > I've just tried the following:
> > > >> > 
> > > >> > call gdalwarp -of GTiff *-wm 13000* -co TILED=YES -co BIGTIFF=YES
> > 
> > -co
> > 
> > > >> > COMPRESS=JPEG -co JPEG_QUALITY=80 -co BLOCKXSIZE=512 -co
> > > >> > BLOCKYSIZE=512
> > > >> 
> > > >> -co
> > > >> 
> > > >> > PHOTOMETRIC=YCBCR SP1937.tif SP1938.tif SP1939.tif 1.tif
> > > >> > 
> > > >> > The documentation says:
> > > >> > > -wm memory_in_mb:
> > > >> > > Set the amount of memory (in megabytes) that the warp API is
> > 
> > allowed
> > 
> > > >> to
> > > >> 
> > > >> > > use for caching.
> > > >> > 
> > > >> > So by my reading I'm allocating a bit less than 13GB (of my
> > > >> > 16,730,672kB RAM) to gdalwarp.
> > > >> > 
> > > >> > Imagine my surprise when I get this error:
> > > >> >  ERROR 5: GDALWarpOptions.Validate()
> > > >> >  
> > > >> >   dfWarpMemoryLimit=13000 is unreasonably small.
> > > >> > 
> > > >> > After more testing, anything with 4 digits works fine (including
> > > >> > 9999),
> > > >> 
> > > >> but
> > > >> 
> > > >> > anything with five digits (i.e. 10,000) gives me that error.
> > > >> > 
> > > >> > Is this a bug?
> > > >> 
> > > >> Rather a undocumented attempt to correct wrong user supplied
> > 
> > parameter.
> > 
> > > >> In gdalwarp utility, if the value of -wm is < 10000, then it is
> > > >> considered as
> > > >> megabytes ( and multiplied by 1024*1024 for GDAL internals),
> > > >> otherwise it is
> > > >> considered as bytes, and passed unmodified to GDAL internals. (the
> > 
> > last
> > 
> > > >> change
> > > >> to that logic was in http://trac.osgeo.org/gdal/changeset/10817 , 7
> > > >> years ago,
> > > >> where the threshold was modified from 4000 to 10000, so at that time
> > 
> > 10
> > 
> > > >> GB was
> > > >> really enormous !)
> > > >> In gdal warper code, there is a test that checks if the memory limit
> > > >> variable
> > > >> (that must be in bytes now) is at least 100000, and bail out if it
> > > >> is not the
> > > >> case.
> > > >> So currently you could specify the size as bytes : "-wm
> > > >> 16730672000".
> > > >> 
> > > >> I'm wondering if we should not remove this GDAL-isense logic in
> > 
> > gdalwarp
> > 
> > > >> utility and just implement documented behaviour... Or maybe document
> > 
> > the
> > 
> > > >> current behaviour.
> > > >> 
> > > >> Note: unless your warping process implies considerable image shape
> > > >> distortion
> > > >> (which is unlikely to be the case here since I don't see any -s_srs
> > > >> or -t_srs
> > > >> flag), you don't need such a huge value. It could decrease
> > > >> performance actually. The only case where it might help is if your
> > > >> source images
> > 
> > are
> > 
> > > >> not
> > > >> tiled.
> > > >> 
> > > >> >  (GDAL 1.10.1)
> > > >> > 
> > > >> > Cheers,
> > > >> > Jonathan
> > > >> 
> > > >> --
> > > >> Geospatial professional services
> > > >> http://even.rouault.free.fr/services.html
> > 
> > --
> > Geospatial professional services
> > http://even.rouault.free.fr/services.html

-- 
Geospatial professional services
http://even.rouault.free.fr/services.html

More information about the gdal-dev mailing list