[gdal-dev] How to control memory usage in gdal and understand caching for local raster data

[Trygve Aspenes]

Hi

I use mapserver which is using gdal for raster from local files. mapserv is
running as processes under fcgi module from apache2
My use case is raster netcdf data in a custom reprojection stored locally
with a week of data every 5 minutes, with new data every 5 minute.
Mapserver, using the raster gdal, reads the data, resample and return the
data. This works fine.

If I test with requesting the same dataset again and again for hundreds of
times, I see little or no memory usage increase of the mapserv process.

However, my setup is datasets with many timestamps. One timestamp per
netcdf file.
If I request each dataset once for each timestamp, I see a steady
increase in memory usage of the mapserv process ( still running under fcgi
module and apache2). I'm guessing this as gdal is caching data. So where
does all this memory usage come from?

I have run valgrind on the mapserv process as a standalone processi. No
leaks.
I have run valgrind on apache2 running the fcgi module running the mapserv
as a long living process. No leaks as far as I can see.

I suspect the gdal block cache mechanism. There is an environmental
variable GDAL_CACHEMAX, which should control the GDAL CACHE usage?

However I have tried to set GDAL_CACHEMAX to various values, even to 0 (
and verified the variable is read with logs output like:

[Thu Feb 26 07:05:34.677597 2026] [fcgid:warn] [pid 421231:tid 421234]
[client ::1:60610] mod_fcgid: stderr: msDrawRasterLayerGDAL():
[Thu Feb 26 07:05:34 2026].6776, 0.0302: GDAL:
GDALDefaultOverviews::OverviewScan()
[Thu Feb 26 07:05:34 2026].6777, 0.0303: LOCK: Lock contention : max = 510,
avg = 510
[Thu Feb 26 07:05:34 2026].6778, 0.0303: GDAL: GDAL_CACHEMAX = 0 MB
[Thu Feb 26 07:05:34 2026].6778, 0.0303: LOCK: Lock contention : max =
6882, avg = 3696

But still, the memory usage just increases, in what I see, in an
uncontrolled manner.

This leads to eventually OOM kills of the mapserv process.

I can of course set the fcgi environmental
variable FcgidMaxRequestsPerProcess to a low value, but that will require
the fcgi module to respawn new mapserv processes all the time.

The data is stored locally, so no vsi is involved as far as I can see.

It must be a way to control the memory usage in gdal in a better way ( if
this really is gdal causing this)

Any ideas?

This test has been set up using mapserver 8.6, gdal 3.12.2 on a ubuntu
noble system.

-- 
Trygve Aspenes - Senioringeniør/Senior Engineer
Telefon/Phone: +47 48223736
Observasjonskvalitet og databehandling (OKD)
Observasjonsdivisjonen/Department of observation
Meteorologisk Institutt /MET Norway
Postboks 43, Blindern 0313 OSLO, NORWAY
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20260226/7826539b/attachment.htm>