[gdal-dev] GDAL python bindings memory usage

[Andrew C Aitchison]

On Thu, 26 Dec 2019, Evert Etienne (SITEMARK) wrote:

> I am measuring memory usage using psutil: Process: 
> `psutil.Process().memory_info().rss` and 
> `psutil.Process().memory_percent()` General: 
> `psutil.virtual_memory().available`
>
> I do tests both on MacOS and linux.
>
> Any idea about how to investigate the file cache or how to use
> psutil to check the difference between this file-system cache and
> other memory usage?

I'm afraid I don't know enough python to help.

On Linux (not sure about MacOS) the command "free" shows this filesystem
cache as "buff/cache".


> By the way, instead of using a real tempfile but removing it and
> using `input + â??.tmpâ??` as output file yields the same behavior.
>
> When calling the gdal calls in a Process, the issues do not appear (3 different approaches)
>
> ```
>    74    109.5 MiB      0.0 MiB           ctx = multiprocessing.get_context('spawn')
>    75    109.5 MiB      0.0 MiB           p = ctx.Process(target=gdalwarp_wrapper, args=(output_path, input_path), kwargs=kwargs)
>    76    109.7 MiB      0.1 MiB           p.start()
>    77    105.5 MiB      0.0 MiB           p.join()
>    78    105.5 MiB      0.0 MiB           ctx = multiprocessing.get_context('fork')
>    79    105.5 MiB      0.0 MiB           p = ctx.Process(target=gdalwarp_wrapper, args=(output_path, input_path), kwargs=kwargs)
>    80    105.5 MiB      0.0 MiB           p.start()
>    81    105.5 MiB      0.0 MiB           p.join()
>    82   3750.6 MiB   3645.0 MiB           gdalwarp_wrapper(output_path, input_path, **kwargs)
> ```
>
> Thanks!
>
> On 26 Dec 2019, at 20:21, Andrew C Aitchison <andrew at aitchison.me.uk<mailto:andrew at aitchison.me.uk>> wrote:
>
> How are you measuring free RAM at this point ?
>
> The GDAL cache will be included in the the memory use reported by
> the python interpreter, but at least on Linux the system command
> "free" also reports the memory used by the file-system cache, which
> will contain (the read parts of) the file, until the system decides
> to use that RAM for something else.  Your tempfile is likely being
> stored in the filesystem cache.