[gdal-dev] Memory access violation with GDALGetRandomRasterSample and NITF/JPEG2000

[Shaw, Jonathan-P29740]

I am using GDALGetRandomRasterSample() to determine scale values for
16-bit per band map sources that I'm processing. What I've found is that
for my NITF files that use JPEG2000 subimages,
GDALGetRandomRasterSample() takes a long time to run and consumes a lot
of memory (~800 MB RAM for a 540 MB NITF). The memory is not released
until the source is closed (with GDALClose()).

One of my sources causes GDAL or the source driver (NITF? JP2ECW?
JP2MrSID?) to commit an access violation by attempting to read or write
addresses 0x0 or 0x4, for instance. This is a larger map (9132x37889
pixels, a 1.2 gigabyte file). Watching in the debugger, I can that
GDALGetRandomRasterSample() takes about 90 seconds or so on the first
raster band. At this point it is using over 900 MB RAM. Maybe 10 seconds
in to processing the second raster band, it crashes with the access
violation (at around a gig of memory usage, maybe less).

The problem can be reproduced by running gdalinfo on the source, with
the -sample option. The -stats option also fails, but only once it
reaches/exceeds 2 gigs memory usage.

Here is what I've attempted so far:

 * FWTools 2.4.7 (1.7.0b2) and 2.2.8 (1.6.0?), on Windows XP x64 (8 gigs
RAM) and in a 32-bit
   virtual machine (2 gigs). My target environment is 32-bit XP.

 * I set GDAL_CACHEMAX to 900 MB, but have also tried 100 MB.

 * I tried setting GDAL_ECW_CACHE_MAXMEM to 200 MB. This appears to make
   no difference whatsoever, so I'm not positive that I'm using the
   JP2ECW driver. "gdalinfo --formats" shows both JP2ECW and JP2MrSID.

 * I've tried opening the large NITF in OpenEV under both 2.4.7 and
2.2.8.
   It takes a while, prompts me to select which subdataset (there are
2),
   I select the subdataset and then it processes for a long time and
   eventually crashes. Basically, it's the same behavior I see from my
   own code.

 * I also tried building GDAL myself from the 1.7.1 sources, but I
cannot
   build the ECW driver because I do not have the required third party
   products. Perhaps because of this, I also could not figure out how to
   step into the GDAL sources for the driver from my VS2008 SP1
debugger.

 * I tried copying the GDALGetRandomRasterSample() into my code and
   stepped through it. The large memory allocations occur when
   GetLockedBlockRef() is called. For these NITF/JPEG2000 files, the
   memory is never released by DropLock().

 * The memory appears to be released when the source dataset is
destructed
   with GDALClose(). This might be a workaround for me, i.e. process one
   band at a time, closing and reopening the file between bands, but I
am
   still concerned about my target platform running out of RAM, and the
   long time that this takes.

 * I'm also considering making a temporary, much smaller GeoTIFF from
the
   source and then computing the scale values from the smaller file. I
   haven't tried this, yet, and would prefer an approach that runs
   quickly with low memory.

Is this a known problem for NITF or JPEG2000 files? Is there a way to
see which JPEG2000 driver is used and to switch it to the other one? Any
other suggestions?

Thanks,
Jonathan