[gdal-dev] -wm (memory in mb with gdalwarp)

[Even Rouault]

Le lundi 02 décembre 2013 19:42:59, Jonathan Moules a écrit :
> On 2 December 2013 18:32, Even Rouault <even.rouault at mines-paris.org> wrote:
> > Le lundi 02 décembre 2013 19:12:07, Jonathan Moules a écrit :
> > > Hi Even,
> > > To hit the 10GB limit I'm using (Windows Server 2008):
> > > 
> > > gdal_translate -of GTiff -co TILED=YES -co BIGTIFF=YES -co
> > > COMPRESS=JPEG -co JPEG_QUALITY=60 -co BLOCKXSIZE=512 -co
> > > BLOCKYSIZE=512 -co
> > > PHOTOMETRIC=YCBCR 2000.vrt 2000.tif
> > > 
> > > I tried adding:  --config GDAL_CACHEMAX 27000000000 but then it'd just
> > > crash with out-of-memory (it wouldn't actually stop at 27GB; so I guess
> > > that works differently again).
> > 
> > Crash (Windows dialog) or "clean" exit with a GDAL error message ?
> > What are the files (TIFF, ECW, ...) in the VRT ?
> > I don't see anything wrong, and have not OS and hardware specs to
> > reproduce that.
> 
> A clean GDAL error:
> 
> 0...ERROR 2: GDALRasterBlock::Internalize : Out of memory allocating 262144

It is really the operating system refusing to do the allocation. Perhaps there 
are too much of small allocation, but the virtual memory space on 64 bit is 
huge so fragmentation should not be an issue.
Or perhaps you've just reached the RAM size due to the ECW SDK eating RAM by 
itself. According to http://www.gdal.org/frmt_ecw.html, it can consume up to 
1/4 of the RAM by default. I'm not sure which version of the SDK you are 
using, but I remember that with the ECW SDK 3.3 it requires a patch for Linux 
64 bit, otherwise it could use much more RAM than 1/4. You could try setting 
ECW_CACHE_MAXMEM explicitely (possibly below 2 GB to avoid any potential 32/64 
bit issue)

> 
> > bytes.
> > ERROR 1: GetBlockRef failed at X block offset 334, Y block offset 60
> 
> The "files" are one single ECW file. 5GB compressed, about 200GB
> uncompressed.

-- 
Geospatial professional services
http://even.rouault.free.fr/services.html