[gdal-dev] memory leak in GRIB reader (with Python bindings)

[Chris Barker]

On Thu, May 24, 2012 at 2:40 PM, Even Rouault
<even.rouault at mines-paris.org> wrote:
> Ok, see http://trac.osgeo.org/gdal/ticket/4682 for a fix. Basically, the
> current caching strategy is maintained (cache all bands that have been
> accessed), until a threshold is reached (arbitrarly set to 100 MB by default).

seems reasonable -- is there an API to change that threshold at run
time? Not I think it's needed...

> When the threshold is reached, then we only cache one band at a time. That
> could be made smarter, but I think this is good enough for now.

it sounds good to me.

meanwhile, closing the dataset every once in a while (in my case,
every 28 bands read) works great -- topping out at around 180MB memory
use.

Thanks for the hint and the fix.

>> maybe -- but what is GDAL policy usually? It doesn't read the data
>> until you ask for it, and I would have expected to keep copy myself if
>> want to use it again.
>
> I'm not a specialist of the GRIB API, but from what I see, it only returns the
> data for a full band, and not for partial reads. So for example, if you
> accessed one line at a time, and that GDAL didn't do any caching, it would
> mean that GDAL would have to decode the whole band each time. Pretty slow !

IIUC, GRIB compresses, so yes, it does make sense to read a whole band at once.

> The previous strategy didn't cache all the bands, but each band once you have
> you read it once. Obviously, if you read all the bands, then at the end, it
> has cached all the bands ;-) The rationale behind this was that it was the
> best strategy to minimize the number of lines of codes in the GRIB driver ;-)

understandable, but sub-optimum!

Thanks,
  -Chris


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov