<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-15">
<meta name="Generator" content="Microsoft Exchange Server">
<!-- converted from text --><style><!-- .EmailQuote { margin-left: 1pt; padding-left: 4pt; border-left: #800000 2px solid; } --></style>
</head>
<body>
<div>
<div dir="auto">The issue is also visible on process level. Do you know if this cache exists in process level and if so, if free can show this as well?</div>
<div class="x_gmail_extra"><br>
<div class="x_gmail_quote">On Dec 27, 2019 20:08, Andrew C Aitchison <andrew@aitchison.me.uk> wrote:<br type="attribution">
</div>
</div>
</div>
<font size="2"><span style="font-size:11pt;">
<div class="PlainText">On Thu, 26 Dec 2019, Evert Etienne (SITEMARK) wrote:<br>
<br>
> I am measuring memory usage using psutil: Process: <br>
> `psutil.Process().memory_info().rss` and <br>
> `psutil.Process().memory_percent()` General: <br>
> `psutil.virtual_memory().available`<br>
><br>
> I do tests both on MacOS and linux.<br>
><br>
> Any idea about how to investigate the file cache or how to use<br>
> psutil to check the difference between this file-system cache and<br>
> other memory usage?<br>
<br>
I'm afraid I don't know enough python to help.<br>
<br>
On Linux (not sure about MacOS) the command "free" shows this filesystem<br>
cache as "buff/cache".<br>
<br>
<br>
> By the way, instead of using a real tempfile but removing it and<br>
> using `input + â??.tmpâ??` as output file yields the same behavior.<br>
><br>
> When calling the gdal calls in a Process, the issues do not appear (3 different approaches)<br>
><br>
> ```<br>
> 74 109.5 MiB 0.0 MiB ctx = multiprocessing.get_context('spawn')<br>
> 75 109.5 MiB 0.0 MiB p = ctx.Process(target=gdalwarp_wrapper, args=(output_path, input_path), kwargs=kwargs)<br>
> 76 109.7 MiB 0.1 MiB p.start()<br>
> 77 105.5 MiB 0.0 MiB p.join()<br>
> 78 105.5 MiB 0.0 MiB ctx = multiprocessing.get_context('fork')<br>
> 79 105.5 MiB 0.0 MiB p = ctx.Process(target=gdalwarp_wrapper, args=(output_path, input_path), kwargs=kwargs)<br>
> 80 105.5 MiB 0.0 MiB p.start()<br>
> 81 105.5 MiB 0.0 MiB p.join()<br>
> 82 3750.6 MiB 3645.0 MiB gdalwarp_wrapper(output_path, input_path, **kwargs)<br>
> ```<br>
><br>
> Thanks!<br>
><br>
> On 26 Dec 2019, at 20:21, Andrew C Aitchison <andrew@aitchison.me.uk<mailto:andrew@aitchison.me.uk>> wrote:<br>
><br>
> How are you measuring free RAM at this point ?<br>
><br>
> The GDAL cache will be included in the the memory use reported by<br>
> the python interpreter, but at least on Linux the system command<br>
> "free" also reports the memory used by the file-system cache, which<br>
> will contain (the read parts of) the file, until the system decides<br>
> to use that RAM for something else. Your tempfile is likely being<br>
> stored in the filesystem cache.<br>
</div>
</span></font>
</body>
</html>