[gdal-dev] GDAL python bindings memory usage

[Andrew C Aitchison]

On Thu, 26 Dec 2019, Evert Etienne (SITEMARK) wrote:

> The smaller caches might indeed explain the difference. Also the ceiling is indeed observed.
>
> Do you have any idea why setting `gdal.SetCacheMax(0)` does not clear the memory?
> Setting it to 0 should explicitly clear the cache according to
> https://github.com/OSGeo/gdal/blob/master/gdal/gcore/gdalrasterblock.cpp#L167
> but `GetCacheUsed` already reports 0 right after the python call
> finishes.
>
> Some more extracts of our logfile:
> ```
> Setting GDAL CacheMax to 31111.349609375 MB
> â?Š
> 95 9702.1 MiB 9593.9 MiB gdal.Warp(temp.name, input_path, **kwargs)
> 97 40839.7 MiB 31137.6 MiB gdal.Translate(output_path, temp.name, creationOptions=copts)
> ```
> We set the CacheMax using the python bindings and we can see the
> Translate call reaches the maximum amount. Afterwards gdal reports
> `gdal cache: 0.0/31111.349609375` (Used/max) but the memory usage
> stays the same.
> To explain why this is a problem for us. Afterwards we run
> gdal2tiles with GDAL_CACHEMAX set to 50% of the free RAM which is
> very low because the cache from the calls before is seen as not-free
> or is not freed.

How are you measuring free RAM at this point ?

The GDAL cache will be included in the the memory use reported by the
python interpreter, but at least on Linux the system command "free"
also reports the memory used by the file-system cache, which will contain
(the read parts of) the file, until the system decides to use that RAM for something else.
Your tempfile is likely being stored in the filesystem cache.

-- 
Andrew C. Aitchison					Kendal, UK
 			andrew at aitchison.me.uk