[gdal-dev] Memory access violation with GDALGetRandomRasterSample
Jonathan.Shaw at gdc4s.com
Thu Mar 4 17:03:10 EST 2010
I am using GDALGetRandomRasterSample() to determine scale values for
16-bit per band map sources that I'm processing. What I've found is that
for my NITF files that use JPEG2000 subimages,
GDALGetRandomRasterSample() takes a long time to run and consumes a lot
of memory (~800 MB RAM for a 540 MB NITF). The memory is not released
until the source is closed (with GDALClose()).
One of my sources causes GDAL or the source driver (NITF? JP2ECW?
JP2MrSID?) to commit an access violation by attempting to read or write
addresses 0x0 or 0x4, for instance. This is a larger map (9132x37889
pixels, a 1.2 gigabyte file). Watching in the debugger, I can that
GDALGetRandomRasterSample() takes about 90 seconds or so on the first
raster band. At this point it is using over 900 MB RAM. Maybe 10 seconds
in to processing the second raster band, it crashes with the access
violation (at around a gig of memory usage, maybe less).
The problem can be reproduced by running gdalinfo on the source, with
the -sample option. The -stats option also fails, but only once it
reaches/exceeds 2 gigs memory usage.
Here is what I've attempted so far:
* FWTools 2.4.7 (1.7.0b2) and 2.2.8 (1.6.0?), on Windows XP x64 (8 gigs
RAM) and in a 32-bit
virtual machine (2 gigs). My target environment is 32-bit XP.
* I set GDAL_CACHEMAX to 900 MB, but have also tried 100 MB.
* I tried setting GDAL_ECW_CACHE_MAXMEM to 200 MB. This appears to make
no difference whatsoever, so I'm not positive that I'm using the
JP2ECW driver. "gdalinfo --formats" shows both JP2ECW and JP2MrSID.
* I've tried opening the large NITF in OpenEV under both 2.4.7 and
It takes a while, prompts me to select which subdataset (there are
I select the subdataset and then it processes for a long time and
eventually crashes. Basically, it's the same behavior I see from my
* I also tried building GDAL myself from the 1.7.1 sources, but I
build the ECW driver because I do not have the required third party
products. Perhaps because of this, I also could not figure out how to
step into the GDAL sources for the driver from my VS2008 SP1
* I tried copying the GDALGetRandomRasterSample() into my code and
stepped through it. The large memory allocations occur when
GetLockedBlockRef() is called. For these NITF/JPEG2000 files, the
memory is never released by DropLock().
* The memory appears to be released when the source dataset is
with GDALClose(). This might be a workaround for me, i.e. process one
band at a time, closing and reopening the file between bands, but I
still concerned about my target platform running out of RAM, and the
long time that this takes.
* I'm also considering making a temporary, much smaller GeoTIFF from
source and then computing the scale values from the smaller file. I
haven't tried this, yet, and would prefer an approach that runs
quickly with low memory.
Is this a known problem for NITF or JPEG2000 files? Is there a way to
see which JPEG2000 driver is used and to switch it to the other one? Any
More information about the gdal-dev