[gdal-dev] -wm (memory in mb with gdalwarp)

[Jonathan Moules]

Thanks for the replies. I think in this instance that as Even suggests it's
because both the ECW SDK and GDAL are trying to allocate which will take me
over the RAM limit.
>From a users perspective I just figured that more RAM = better, based on
experience that's usually how things work. Useful to know it's not here.

Thanks again,
Jonathan

On 2 December 2013 19:03, Even Rouault <even.rouault at mines-paris.org> wrote:

> Le lundi 02 décembre 2013 19:42:59, Jonathan Moules a écrit :
> > On 2 December 2013 18:32, Even Rouault <even.rouault at mines-paris.org>
> wrote:
> > > Le lundi 02 décembre 2013 19:12:07, Jonathan Moules a écrit :
> > > > Hi Even,
> > > > To hit the 10GB limit I'm using (Windows Server 2008):
> > > >
> > > > gdal_translate -of GTiff -co TILED=YES -co BIGTIFF=YES -co
> > > > COMPRESS=JPEG -co JPEG_QUALITY=60 -co BLOCKXSIZE=512 -co
> > > > BLOCKYSIZE=512 -co
> > > > PHOTOMETRIC=YCBCR 2000.vrt 2000.tif
> > > >
> > > > I tried adding:  --config GDAL_CACHEMAX 27000000000 but then it'd
> just
> > > > crash with out-of-memory (it wouldn't actually stop at 27GB; so I
> guess
> > > > that works differently again).
> > >
> > > Crash (Windows dialog) or "clean" exit with a GDAL error message ?
> > > What are the files (TIFF, ECW, ...) in the VRT ?
> > > I don't see anything wrong, and have not OS and hardware specs to
> > > reproduce that.
> >
> > A clean GDAL error:
> >
> > 0...ERROR 2: GDALRasterBlock::Internalize : Out of memory allocating
> 262144
>
> It is really the operating system refusing to do the allocation. Perhaps
> there
> are too much of small allocation, but the virtual memory space on 64 bit is
> huge so fragmentation should not be an issue.
> Or perhaps you've just reached the RAM size due to the ECW SDK eating RAM
> by
> itself. According to http://www.gdal.org/frmt_ecw.html, it can consume up
> to
> 1/4 of the RAM by default. I'm not sure which version of the SDK you are
> using, but I remember that with the ECW SDK 3.3 it requires a patch for
> Linux
> 64 bit, otherwise it could use much more RAM than 1/4. You could try
> setting
> ECW_CACHE_MAXMEM explicitely (possibly below 2 GB to avoid any potential
> 32/64
> bit issue)
>
> >
> > > bytes.
> > > ERROR 1: GetBlockRef failed at X block offset 334, Y block offset 60
> >
> > The "files" are one single ECW file. 5GB compressed, about 200GB
> > uncompressed.
>
> --
> Geospatial professional services
> http://even.rouault.free.fr/services.html
>

-- 
This transmission is intended for the named addressee(s) only and may 
contain sensitive or protectively marked material up to RESTRICTED and 
should be handled accordingly. Unless you are the named addressee (or 
authorised to receive it for the addressee) you may not copy or use it, or 
disclose it to anyone else. If you have received this transmission in error 
please notify the sender immediately. All email traffic sent to or from us, 
including without limitation all GCSX traffic, may be subject to recording 
and/or monitoring in accordance with relevant legislation.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20131203/4558683b/attachment.html>