[gdal-dev] GDAL python bindings memory usage

[Evert Etienne (SITEMARK)]

The smaller caches might indeed explain the difference. Also the ceiling is indeed observed.

Do you have any idea why setting `gdal.SetCacheMax(0)` does not clear the memory?
Setting it to 0 should explicitly clear the cache according to https://github.com/OSGeo/gdal/blob/master/gdal/gcore/gdalrasterblock.cpp#L167 but `GetCacheUsed` already reports 0 right after the python call finishes.

Some more extracts of our logfile:
```
Setting GDAL CacheMax to 31111.349609375 MB
…
95 9702.1 MiB 9593.9 MiB gdal.Warp(temp.name, input_path, **kwargs)
97 40839.7 MiB 31137.6 MiB gdal.Translate(output_path, temp.name, creationOptions=copts)
```
We set the CacheMax using the python bindings and we can see the Translate call reaches the maximum amount. Afterwards gdal reports `gdal cache: 0.0/31111.349609375` (Used/max) but the memory usage stays the same.
To explain why this is a problem for us. Afterwards we run gdal2tiles with GDAL_CACHEMAX set to 50% of the free RAM which is very low because the cache from the calls before is seen as not-free or is not freed.

Thanks already for helping me look into this.

Evert

On 26 Dec 2019, at 16:07, Even Rouault <even.rouault at spatialys.com<mailto:even.rouault at spatialys.com>> wrote:

On jeudi 26 décembre 2019 14:30:35 CET Evert Etienne (SITEMARK) wrote:
Using a bigger file (8GB) and a machine with 64GB Ram we can see the
increase being higher. For both gdal.Warp and gdal.Translate

GDAL may use up to GDAL_CACHEMAX memory, plus some other smaller caches. I'm
not sure how memory_profile monitors memory consumption, but it is possible
that the apparent virtual memory is not returned to the OS. But if you run
those commands several times in the same procss, you should hopefully observe
some ceiling in memory consumption.

Even

--
Spatialys - Geospatial professional services
https://eur03.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.spatialys.com&data=01%7C01%7Cevert.etienne%40sitemark.com%7Cb0c5f46254fc4bad08ac08d78a15516c%7Cfc89adff07ac47008853b7b7e906068e%7C0&sdata=St%2FX27v2p7nYw2IP2RwKMnH3ghGSJFj8Jfld5%2F%2B%2BYNY%3D&reserved=0

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20191226/9801c5b2/attachment.html>