[gdal-dev] GDAL python bindings memory usage

[Evert Etienne (SITEMARK)]

I am measuring memory usage using psutil:
Process: `psutil.Process().memory_info().rss` and `psutil.Process().memory_percent()`
General: `psutil.virtual_memory().available`

I do tests both on MacOS and linux.

Any idea about how to investigate the file cache or how to use psutil to check the difference between this file-system cache and other memory usage?
By the way, instead of using a real tempfile but removing it and using `input + ’.tmp’` as output file yields the same behavior.

When calling the gdal calls in a Process, the issues do not appear (3 different approaches)

```
    74    109.5 MiB      0.0 MiB           ctx = multiprocessing.get_context('spawn')
    75    109.5 MiB      0.0 MiB           p = ctx.Process(target=gdalwarp_wrapper, args=(output_path, input_path), kwargs=kwargs)
    76    109.7 MiB      0.1 MiB           p.start()
    77    105.5 MiB      0.0 MiB           p.join()
    78    105.5 MiB      0.0 MiB           ctx = multiprocessing.get_context('fork')
    79    105.5 MiB      0.0 MiB           p = ctx.Process(target=gdalwarp_wrapper, args=(output_path, input_path), kwargs=kwargs)
    80    105.5 MiB      0.0 MiB           p.start()
    81    105.5 MiB      0.0 MiB           p.join()
    82   3750.6 MiB   3645.0 MiB           gdalwarp_wrapper(output_path, input_path, **kwargs)
```

Thanks!

On 26 Dec 2019, at 20:21, Andrew C Aitchison <andrew at aitchison.me.uk<mailto:andrew at aitchison.me.uk>> wrote:

How are you measuring free RAM at this point ?

The GDAL cache will be included in the the memory use reported by the
python interpreter, but at least on Linux the system command "free"
also reports the memory used by the file-system cache, which will contain
(the read parts of) the file, until the system decides to use that RAM for something else.
Your tempfile is likely being stored in the filesystem cache.

--
Andrew C. Aitchison Kendal, UK
andrew at aitchison.me.uk<mailto:andrew at aitchison.me.uk>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20191226/4ed75fae/attachment.html>