[gdal-dev] -wm (memory in mb with gdalwarp)

Mon Dec 2 10:12:07 PST 2013

Hi Even,
To hit the 10GB limit I'm using (Windows Server 2008):

gdal_translate -of GTiff -co TILED=YES -co BIGTIFF=YES -co COMPRESS=JPEG
-co JPEG_QUALITY=60 -co BLOCKXSIZE=512 -co BLOCKYSIZE=512 -co
PHOTOMETRIC=YCBCR 2000.vrt 2000.tif

I tried adding:  --config GDAL_CACHEMAX 27000000000 but then it'd just
crash with out-of-memory (it wouldn't actually stop at 27GB; so I guess
that works differently again).

====

I was also using this (which is a simple tif to tif) and it too used 10GB
before stopping itself (this was Windows 7):
call gdal_merge -o output.tif -of GTiff -co TILED=YES -co BIGTIFF=YES -co
COMPRESS=JPEG -co JPEG_QUALITY=80 -co BLOCKXSIZE=512 -co BLOCKYSIZE=512
--optfile tiff_list.txt -co PHOTOMETRIC=YCBCR

It may just have been co-incident.

I think I'll just let it do it automatically and hope for the best.

Jonathan

On 2 December 2013 17:57, Even Rouault <even.rouault at mines-paris.org> wrote:

> Le lundi 02 décembre 2013 18:49:42, Jonathan Moules a écrit :
> > Hi Even,
> > A further thought while we're on the issue of limits, it appears that
> gdal
> > (both gdal_translate and gdalwarp) have an internal limit of 10GB when
> > they're auto-allocating RAM.
>
> I'm not aware of such a limitation in the code. It might be that your OS
> cannot allocate that much memory in one single malloc() if it is the
> allocation pattern involved here, but I'm just guessing. What commands did
> you
> exactly tried to trigger this ?
>
> >
> > Assuming this is the case and it's not just coincidence, it might it be
> > worth updating them to allow more use if appropriate (it seems to be
> > hitting this for a large-ecw to geotiff convesion)?
>
> The ECW SDK itself has its own RAM management. See
> http://www.gdal.org/frmt_ecw.html
>
> > For instance my work
> > desktop has 16GB and the server I'm also doing some processing on has
> 36GB.
> >
> > Cheers,
> > Jonathan
> >
> > (also I failed to reply-all to my previous reply; so included below)
> >
> >
> > On 2 December 2013 17:25, Jonathan Moules <
> >
> > jonathanmoules at warwickshire.gov.uk> wrote:
> > > Hi Even,
> > > Thanks for the detailed reply. It seems in the end that I can use a
> > > combination of gdalbuildvrt and gdal_translate to get the same results
> > > without having to bother with manual memory management. The output is
> the
> > > same size and the quality is identical.
> > >
> > > From a user perspective, I would suggest removing this behaviour
> (though
> > > documenting would work too). You could add a gdal warning if they set
> it
> > > above 10GB with a similar caveat to the "Note" in your email.
> > > Of course, then in another 7 years you'll be asked why the warning is
> > > triggering at the "low" 10GB!
> > >
> > > Thanks again,
> > > Jonathan
> > >
> > > On 2 December 2013 16:39, Even Rouault <even.rouault at mines-
> paris.org>wrote:
> > >> Le lundi 02 décembre 2013 17:17:11, Jonathan Moules a écrit :
> > >> > Hi List,
> > >> > I've just tried the following:
> > >> >
> > >> > call gdalwarp -of GTiff *-wm 13000* -co TILED=YES -co BIGTIFF=YES
> -co
> > >> > COMPRESS=JPEG -co JPEG_QUALITY=80 -co BLOCKXSIZE=512 -co
> > >> > BLOCKYSIZE=512
> > >>
> > >> -co
> > >>
> > >> > PHOTOMETRIC=YCBCR SP1937.tif SP1938.tif SP1939.tif 1.tif
> > >> >
> > >> > The documentation says:
> > >> > > -wm memory_in_mb:
> > >> > > Set the amount of memory (in megabytes) that the warp API is
> allowed
> > >>
> > >> to
> > >>
> > >> > > use for caching.
> > >> >
> > >> > So by my reading I'm allocating a bit less than 13GB (of my
> > >> > 16,730,672kB RAM) to gdalwarp.
> > >> >
> > >> > Imagine my surprise when I get this error:
> > >> >  ERROR 5: GDALWarpOptions.Validate()
> > >> >
> > >> >   dfWarpMemoryLimit=13000 is unreasonably small.
> > >> >
> > >> > After more testing, anything with 4 digits works fine (including
> > >> > 9999),
> > >>
> > >> but
> > >>
> > >> > anything with five digits (i.e. 10,000) gives me that error.
> > >> >
> > >> > Is this a bug?
> > >>
> > >> Rather a undocumented attempt to correct wrong user supplied
> parameter.
> > >>
> > >> In gdalwarp utility, if the value of -wm is < 10000, then it is
> > >> considered as
> > >> megabytes ( and multiplied by 1024*1024 for GDAL internals), otherwise
> > >> it is
> > >> considered as bytes, and passed unmodified to GDAL internals. (the
> last
> > >> change
> > >> to that logic was in http://trac.osgeo.org/gdal/changeset/10817 , 7
> > >> years ago,
> > >> where the threshold was modified from 4000 to 10000, so at that time
> 10
> > >> GB was
> > >> really enormous !)
> > >> In gdal warper code, there is a test that checks if the memory limit
> > >> variable
> > >> (that must be in bytes now) is at least 100000, and bail out if it is
> > >> not the
> > >> case.
> > >> So currently you could specify the size as bytes : "-wm 16730672000".
> > >>
> > >> I'm wondering if we should not remove this GDAL-isense logic in
> gdalwarp
> > >> utility and just implement documented behaviour... Or maybe document
> the
> > >> current behaviour.
> > >>
> > >> Note: unless your warping process implies considerable image shape
> > >> distortion
> > >> (which is unlikely to be the case here since I don't see any -s_srs or
> > >> -t_srs
> > >> flag), you don't need such a huge value. It could decrease performance
> > >> actually. The only case where it might help is if your source images
> are
> > >> not
> > >> tiled.
> > >>
> > >> >  (GDAL 1.10.1)
> > >> >
> > >> > Cheers,
> > >> > Jonathan
> > >>
> > >> --
> > >> Geospatial professional services
> > >> http://even.rouault.free.fr/services.html
>
> --
> Geospatial professional services
> http://even.rouault.free.fr/services.html
>

-- 
This transmission is intended for the named addressee(s) only and may 
contain sensitive or protectively marked material up to RESTRICTED and 
should be handled accordingly. Unless you are the named addressee (or 
authorised to receive it for the addressee) you may not copy or use it, or 
disclose it to anyone else. If you have received this transmission in error 
please notify the sender immediately. All email traffic sent to or from us, 
including without limitation all GCSX traffic, may be subject to recording 
and/or monitoring in accordance with relevant legislation.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20131202/27d8196b/attachment-0001.html>