[gdal-dev] GDAL python bindings memory usage

[Evert Etienne (SITEMARK)]

The issue is also visible on process level. Do you know if this cache exists in process level and if so, if free can show this as well?

On Dec 27, 2019 20:08, Andrew C Aitchison <andrew at aitchison.me.uk> wrote:
On Thu, 26 Dec 2019, Evert Etienne (SITEMARK) wrote:

> I am measuring memory usage using psutil: Process:
> `psutil.Process().memory_info().rss` and
> `psutil.Process().memory_percent()` General:
> `psutil.virtual_memory().available`
>
> I do tests both on MacOS and linux.
>
> Any idea about how to investigate the file cache or how to use
> psutil to check the difference between this file-system cache and
> other memory usage?

I'm afraid I don't know enough python to help.

On Linux (not sure about MacOS) the command "free" shows this filesystem
cache as "buff/cache".


> By the way, instead of using a real tempfile but removing it and
> using `input + â??.tmpâ??` as output file yields the same behavior.
>
> When calling the gdal calls in a Process, the issues do not appear (3 different approaches)
>
> ```
>    74    109.5 MiB      0.0 MiB           ctx = multiprocessing.get_context('spawn')
>    75    109.5 MiB      0.0 MiB           p = ctx.Process(target=gdalwarp_wrapper, args=(output_path, input_path), kwargs=kwargs)
>    76    109.7 MiB      0.1 MiB           p.start()
>    77    105.5 MiB      0.0 MiB           p.join()
>    78    105.5 MiB      0.0 MiB           ctx = multiprocessing.get_context('fork')
>    79    105.5 MiB      0.0 MiB           p = ctx.Process(target=gdalwarp_wrapper, args=(output_path, input_path), kwargs=kwargs)
>    80    105.5 MiB      0.0 MiB           p.start()
>    81    105.5 MiB      0.0 MiB           p.join()
>    82   3750.6 MiB   3645.0 MiB           gdalwarp_wrapper(output_path, input_path, **kwargs)
> ```
>
> Thanks!
>
> On 26 Dec 2019, at 20:21, Andrew C Aitchison <andrew at aitchison.me.uk<mailto:andrew at aitchison.me.uk>> wrote:
>
> How are you measuring free RAM at this point ?
>
> The GDAL cache will be included in the the memory use reported by
> the python interpreter, but at least on Linux the system command
> "free" also reports the memory used by the file-system cache, which
> will contain (the read parts of) the file, until the system decides
> to use that RAM for something else.  Your tempfile is likely being
> stored in the filesystem cache.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20191227/5a3a5dca/attachment-0001.html>