<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
The smaller caches might indeed explain the difference. Also the ceiling is indeed observed.
<div class=""><br class="">
</div>
<div class="">Do you have any idea why setting `<span style="color: rgb(74, 92, 110); font-family: Menlo; background-color: rgb(255, 255, 255);" class="">gdal.SetCacheMax(</span><span style="font-family: Menlo; background-color: rgb(255, 255, 255); color: rgb(93, 151, 196);" class="">0</span><span style="color: rgb(74, 92, 110); font-family: Menlo; background-color: rgb(255, 255, 255);" class="">)</span>`
does not clear the memory?
<div class="">Setting it to 0 should explicitly clear the cache according to <a href="https://github.com/OSGeo/gdal/blob/master/gdal/gcore/gdalrasterblock.cpp#L167" class="">https://github.com/OSGeo/gdal/blob/master/gdal/gcore/gdalrasterblock.cpp#L167</a> but
`<span style="color: rgb(74, 92, 110); font-family: Menlo; background-color: rgb(255, 255, 255);" class="">GetCacheUsed</span><span style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);" class="">` already reports 0 right after the python call finishes. </span></div>
<div class=""><span style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);" class=""><br class="">
</span></div>
<div class=""><span style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);" class="">Some more extracts of our logfile:</span></div>
<div class=""><span style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);" class="">```</span></div>
<div class="">
<div class="" style="box-sizing: border-box; padding: 0px; margin: 0px; outline: 0px; color: rgb(0, 0, 0); font-family: "Proxima Nova", sans-serif; font-size: 13px; font-variant-ligatures: normal; orphans: 2; white-space: nowrap; widows: 2;">
Setting GDAL CacheMax to 31111.349609375 MB</div>
</div>
<div class="">…</div>
<div class="">
<div class="" style="box-sizing: border-box; padding: 0px; margin: 0px; color: rgb(0, 0, 0); font-family: "Proxima Nova", sans-serif; font-size: 13px; font-variant-ligatures: normal; orphans: 2; white-space: nowrap; widows: 2; outline: none !important;">
95 9702.1 MiB 9593.9 MiB gdal.Warp(temp.name, input_path, **kwargs)</div>
<div class="" style="box-sizing: border-box; padding: 0px; margin: 0px; color: rgb(0, 0, 0); font-family: "Proxima Nova", sans-serif; font-size: 13px; font-variant-ligatures: normal; orphans: 2; white-space: nowrap; widows: 2; outline: none !important;">
97 40839.7 MiB 31137.6 MiB gdal.Translate(output_path, temp.name, creationOptions=copts)</div>
</div>
<div class="">```</div>
<div class="">We set the <span style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); font-family: "Proxima Nova", sans-serif; font-size: 13px; orphans: 2; white-space: nowrap; widows: 2;" class="">CacheMax using the python bindings and we can see the Translate
call reaches the maximum amount. Afterwards gdal reports `</span><span style="color: rgb(0, 0, 0); font-family: "Proxima Nova", sans-serif; font-size: 13px; orphans: 2; white-space: nowrap; widows: 2; background-color: rgb(255, 255, 255);" class="">gdal cache:
0.0/31111.349609375` (Used/max) </span><span style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); font-family: "Proxima Nova", sans-serif; font-size: 13px; orphans: 2; white-space: nowrap; widows: 2;" class="">but the memory usage stays the same. </span></div>
<div class=""><span style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); font-family: "Proxima Nova", sans-serif; font-size: 13px; orphans: 2; white-space: nowrap; widows: 2;" class="">To explain why this is a problem for us. Afterwards we run gdal2tiles
with GDAL_CACHEMAX set to 50% of the free RAM which is very low because the cache from the calls before is seen as not-free or is not freed.</span></div>
<div class=""><span style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); font-family: "Proxima Nova", sans-serif; font-size: 13px; orphans: 2; white-space: nowrap; widows: 2;" class=""><br class="">
</span></div>
<div class=""><span style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); font-family: "Proxima Nova", sans-serif; font-size: 13px; orphans: 2; white-space: nowrap; widows: 2;" class="">Thanks already for helping me look into this.</span></div>
<div class=""><span style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); font-family: "Proxima Nova", sans-serif; font-size: 13px; orphans: 2; white-space: nowrap; widows: 2;" class=""><br class="">
</span></div>
<div class=""><span style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); font-family: "Proxima Nova", sans-serif; font-size: 13px; orphans: 2; white-space: nowrap; widows: 2;" class="">Evert</span></div>
<div class=""><font color="#4a5c6e" face="Menlo" class=""><span style="caret-color: rgb(74, 92, 110); background-color: rgb(255, 255, 255);" class=""><br class="">
</span></font>
<div>
<blockquote type="cite" class="">
<div class="">On 26 Dec 2019, at 16:07, Even Rouault <<a href="mailto:even.rouault@spatialys.com" class="">even.rouault@spatialys.com</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div class="">On jeudi 26 décembre 2019 14:30:35 CET Evert Etienne (SITEMARK) wrote:<br class="">
<blockquote type="cite" class="">Using a bigger file (8GB) and a machine with 64GB Ram we can see the<br class="">
increase being higher. For both gdal.Warp and gdal.Translate<br class="">
</blockquote>
<br class="">
GDAL may use up to GDAL_CACHEMAX memory, plus some other smaller caches. I'm <br class="">
not sure how memory_profile monitors memory consumption, but it is possible <br class="">
that the apparent virtual memory is not returned to the OS. But if you run <br class="">
those commands several times in the same procss, you should hopefully observe <br class="">
some ceiling in memory consumption.<br class="">
<br class="">
Even<br class="">
<br class="">
-- <br class="">
Spatialys - Geospatial professional services<br class="">
<a href="https://eur03.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.spatialys.com&data=01%7C01%7Cevert.etienne%40sitemark.com%7Cb0c5f46254fc4bad08ac08d78a15516c%7Cfc89adff07ac47008853b7b7e906068e%7C0&sdata=St%2FX27v2p7nYw2IP2RwKMnH3ghGSJFj8Jfld5%2F%2B%2BYNY%3D&reserved=0" class="">https://eur03.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.spatialys.com&data=01%7C01%7Cevert.etienne%40sitemark.com%7Cb0c5f46254fc4bad08ac08d78a15516c%7Cfc89adff07ac47008853b7b7e906068e%7C0&sdata=St%2FX27v2p7nYw2IP2RwKMnH3ghGSJFj8Jfld5%2F%2B%2BYNY%3D&reserved=0</a><br class="">
</div>
</div>
</blockquote>
</div>
<br class="">
</div>
</div>
</body>
</html>