<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
I am measuring memory usage using psutil:
<div class="">Process: `<span style="color: rgb(74, 92, 110); font-family: Menlo; background-color: rgb(255, 255, 255);" class="">psutil.Process().memory_info().rss</span><span style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);" class="">` and `</span><span style="color: rgb(74, 92, 110); font-family: Menlo; background-color: rgb(255, 255, 255);" class="">psutil.Process().memory_percent()</span><span style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);" class="">`</span></div>
<div class=""><span style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);" class="">General: `</span><span style="color: rgb(74, 92, 110); font-family: Menlo; background-color: rgb(255, 255, 255);" class="">psutil.virtual_memory().available</span><span style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);" class="">`</span></div>
<div class=""><br class="">
</div>
<div>I do tests both on MacOS and linux.</div>
<div><br class="">
</div>
<div>Any idea about how to investigate the file cache or how to use psutil to check the difference between this file-system cache and other memory usage?</div>
<div>By the way, instead of using a real tempfile but removing it and using `input + ’.tmp’` as output file yields the same behavior.</div>
<div><br class="">
</div>
<div>When calling the gdal calls in a Process, the issues do not appear (3 different approaches)</div>
<div><br class="">
</div>
<div>```</div>
<div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Helvetica Neue";" class="">
74 109.5 MiB 0.0 MiB ctx = multiprocessing.get_context('spawn')</div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Helvetica Neue";" class="">
75 109.5 MiB 0.0 MiB p = ctx.Process(target=gdalwarp_wrapper, args=(output_path, input_path), kwargs=kwargs)</div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Helvetica Neue";" class="">
76 109.7 MiB 0.1 MiB p.start()</div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Helvetica Neue";" class="">
77 105.5 MiB 0.0 MiB p.join()</div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Helvetica Neue";" class="">
78 105.5 MiB 0.0 MiB ctx = multiprocessing.get_context('fork')</div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Helvetica Neue";" class="">
79 105.5 MiB 0.0 MiB p = ctx.Process(target=gdalwarp_wrapper, args=(output_path, input_path), kwargs=kwargs)</div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Helvetica Neue";" class="">
80 105.5 MiB 0.0 MiB p.start()</div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Helvetica Neue";" class="">
81 105.5 MiB 0.0 MiB p.join()</div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Helvetica Neue";" class="">
82 3750.6 MiB 3645.0 MiB gdalwarp_wrapper(output_path, input_path, **kwargs)</div>
<div class="">```</div>
</div>
<div><br class="">
</div>
<div>Thanks!</div>
<div><br class="">
<blockquote type="cite" class="">
<div class="">On 26 Dec 2019, at 20:21, Andrew C Aitchison <<a href="mailto:andrew@aitchison.me.uk" class="">andrew@aitchison.me.uk</a>> wrote:</div>
<div class="">
<div class=""><br class="">
How are you measuring free RAM at this point ?<br class="">
<br class="">
The GDAL cache will be included in the the memory use reported by the<br class="">
python interpreter, but at least on Linux the system command "free"<br class="">
also reports the memory used by the file-system cache, which will contain<br class="">
(the read parts of) the file, until the system decides to use that RAM for something else.<br class="">
Your tempfile is likely being stored in the filesystem cache.<br class="">
<br class="">
-- <br class="">
Andrew C. Aitchison<span class="Apple-tab-span" style="white-space:pre"> </span><span class="Apple-tab-span" style="white-space:pre"></span><span class="Apple-tab-span" style="white-space:pre"></span><span class="Apple-tab-span" style="white-space:pre"></span><span class="Apple-tab-span" style="white-space:pre"></span>Kendal,
UK<br class="">
<span class="Apple-tab-span" style="white-space:pre"></span><span class="Apple-tab-span" style="white-space:pre"></span><span class="Apple-tab-span" style="white-space:pre"></span><a href="mailto:andrew@aitchison.me.uk" class="">andrew@aitchison.me.uk</a></div>
</div>
</blockquote>
</div>
<br class="">
</body>
</html>