<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
Using a bigger file (8GB) and a machine with 64GB Ram we can see the increase being higher. For both gdal.Warp and gdal.Translate
<div class=""><br class="">
</div>
<div class="">```</div>
<div class="">
<pre style="color: rgb(0, 0, 0); font-variant-ligatures: normal; orphans: 2; widows: 2; position: absolute; top: -99px;" class=""> 97 112.2 MiB 0.0 MiB logging.debug(kwargs)
98 691.5 MiB 579.3 MiB gdal.Warp(temp.name, input_path, **kwargs)
99 691.5 MiB 0.0 MiB logging.debug('Compressing image...')
100 3943.1 MiB 3251.6 MiB gdal.Translate(output_path, temp.name, creationOptions=copts, callback=progress_logging('Compressing image', one_is_max=True))</pre>
<div class="">
<pre style="background-color: rgb(255, 255, 255); color: rgb(74, 92, 110); font-family: Menlo;" class=""> 97 112.2 MiB 0.0 MiB logging.debug(kwargs)<br class=""> 98 691.5 MiB 579.3 MiB gdal.Warp(temp.name, input_path, **kwargs)<br class="">100 3943.1 MiB 3251.6 MiB gdal.Translate(output_path, temp.name, creationOptions=copts)</pre>
<div class="">```</div>
</div>
<div><br class="">
<blockquote type="cite" class="">
<div class="">On 26 Dec 2019, at 15:26, Evert Etienne (SITEMARK) <<a href="mailto:evert.etienne@sitemark.com" class="">evert.etienne@sitemark.com</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<span style="caret-color: rgb(0, 0, 0);" class="">Hi all,</span>
<div class="" style="caret-color: rgb(0, 0, 0);"><br class="">
</div>
<div class="" style="caret-color: rgb(0, 0, 0);">I have a question about memory usage of the python gdal bindings. For some GDAL calls (python or not), we try to optimise the gdal cache. Doing this, I’ve noticed the free RAM decreasing after doing gdal operations.
I have been able to narrow it down to the python bindings. Using `memory_profiler` (<a href="https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpypi.org%2Fproject%2Fmemory-profiler%2F&data=01%7C01%7Cevert.etienne%40sitemark.com%7C341fa1258c3c4f700a4e08d78a0f8e45%7Cfc89adff07ac47008853b7b7e906068e%7C0&sdata=PxqBDpBlLOr8eiUQXw9fSdSfCH8lKnUeLLCbciVMO5E%3D&reserved=0" originalsrc="https://pypi.org/project/memory-profiler/" shash="CnbLDAAnyyeidrqFiQC1KwW1QNXeWkQ0b4YtPRbz0yYvjoPgToNPdbhqH1Cwzsr3V/sfPJohewxiRYuXM3jgi9EeOayIz3NVI9OJdfBNZI6/4l186255DmjIw6u3gId0h39kJNEn46pJtrPynoavLOc+rTq2jNyGKcFUXH8afa8=" class="">https://pypi.org/project/memory-profiler/</a>)
I get the following:</div>
<div class="" style="caret-color: rgb(0, 0, 0);">
<div id="line-by-line-memory-usage" class="" style="box-sizing: inherit; color: rgb(70, 70, 70); font-family: "Source Sans Pro", Helvetica, Arial, sans-serif; font-size: 15.2px; font-variant-ligatures: normal; orphans: 2; widows: 2; background-color: rgb(253, 253, 253);">
<p class="" style="box-sizing: inherit; margin: 15px 0px 0px; padding: 0px;">The first column represents the line number of the code that has been profiled, the second column (<span class="" style="box-sizing: inherit; font-style: italic;">Mem usage</span>)
the memory usage of the Python interpreter after that line has been executed. The third column (<span class="" style="box-sizing: inherit; font-style: italic;">Increment</span>) represents the difference in memory of the current line with respect to the last
one. The last column (<span class="" style="box-sizing: inherit; font-style: italic;">Line Contents</span>) prints the code that has been profiled.</p>
</div>
<div id="decorator" class="" style="box-sizing: inherit; color: rgb(70, 70, 70); font-family: "Source Sans Pro", Helvetica, Arial, sans-serif; font-size: 15.2px; font-variant-ligatures: normal; orphans: 2; widows: 2; background-color: rgb(253, 253, 253);">
</div>
</div>
<div class="" style="caret-color: rgb(0, 0, 0);"><br class="">
</div>
<div class="" style="caret-color: rgb(0, 0, 0);">```</div>
<div class="" style="caret-color: rgb(0, 0, 0);">
<div class="">101 65.4 MiB 0.0 MiB logging.debug(kwargs)</div>
<div class="">102 203.9 MiB 138.4 MiB gdal.Warp(temp.name, input_path, **kwargs)</div>
</div>
<div class="" style="caret-color: rgb(0, 0, 0);">```</div>
<div class="" style="caret-color: rgb(0, 0, 0);"><br class="">
</div>
<div class="" style="caret-color: rgb(0, 0, 0);">It does seem related to the cache because of the following tests, but only partially. I would expect since every file is on disk that these calls do not have any lasting effect on memory usage.</div>
<div class="" style="caret-color: rgb(0, 0, 0);"><br class="">
</div>
<div class="" style="caret-color: rgb(0, 0, 0);">```</div>
<div class="" style="caret-color: rgb(0, 0, 0);">
<div class="">98 65.4 MiB 0.0 MiB gdal.SetCacheMax(0)</div>
<div class="">99 87.8 MiB 22.4 MiB gdal.Warp(temp.name, input_path, **kwargs)</div>
</div>
<div class="" style="caret-color: rgb(0, 0, 0);">```</div>
<div class="" style="caret-color: rgb(0, 0, 0);"><br class="">
</div>
<div class="" style="caret-color: rgb(0, 0, 0);">temp.name is a `<span class="" style="color: rgb(74, 92, 110); font-family: Menlo; background-color: rgb(255, 255, 255);">tempfile.NamedTemporaryFile(</span><span class="" style="font-family: Menlo; background-color: rgb(255, 255, 255); color: rgb(166, 202, 145);">'w+</span><span class="" style="background-color: rgb(255, 255, 255);"><font color="#a6ca91" face="Menlo" class=""><span class="" style="caret-color: rgb(166, 202, 145);">’</span></font><font color="#4a5c6e" face="Menlo" class="">)</font></span>`
(`/var/folders/3t/_j9hh3_907g646cgt8pkkjch0000gn/T/tmpumywovz7`. The passed kwargs are ` {'dstSRS': 'EPSG:3857', 'resampleAlg': 2, 'format': 'gtiff', 'multithread': True, 'warpOptions': ['NUM_THREADS=ALL_CPUS'], 'creationOptions': ['BIGTIFF=YES', 'NUM_THREADS=ALL_CPUS’]}`.
The input file is 84.5 MB.</div>
<div class="" style="caret-color: rgb(0, 0, 0);"><br class="">
</div>
<div class="" style="caret-color: rgb(0, 0, 0);">Assigning and deleting the result does not affect the results. They grow bigger but also decrease after deletion. I assume this is the dataset size.</div>
<div class="" style="caret-color: rgb(0, 0, 0);"><br class="">
</div>
<div class="" style="caret-color: rgb(0, 0, 0);">```</div>
<div class="" style="caret-color: rgb(0, 0, 0);">
<div class=""> 96 65.4 MiB 0.0 MiB logging.debug(kwargs)</div>
<div class=""> 97 249.8 MiB 184.4 MiB ds = gdal.Warp(temp.name, input_path, **kwargs)</div>
<div class=""> 98 193.8 MiB 0.0 MiB del ds</div>
<div class="">```</div>
</div>
<div class="" style="caret-color: rgb(0, 0, 0);"><br class="">
</div>
<div class="" style="caret-color: rgb(0, 0, 0);">Am I overlooking any cause for this memory increase or is there a possibility to clear this?</div>
<div class="" style="caret-color: rgb(0, 0, 0);">Am I correct to assume the usage of the gdal python bindings in this way (All files are on disk) should have barely any effect on script memory usage?</div>
<div class="" style="caret-color: rgb(0, 0, 0);"><br class="">
</div>
<div class="" style="caret-color: rgb(0, 0, 0);">Thanks in advance.</div>
</div>
_______________________________________________<br class="">
gdal-dev mailing list<br class="">
<a href="mailto:gdal-dev@lists.osgeo.org" class="">gdal-dev@lists.osgeo.org</a><br class="">
https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.osgeo.org%2Fmailman%2Flistinfo%2Fgdal-dev&data=01%7C01%7Cevert.etienne%40sitemark.com%7C341fa1258c3c4f700a4e08d78a0f8e45%7Cfc89adff07ac47008853b7b7e906068e%7C0&sdata=swgZAj2FYOzIEkzJo6%2FlDaeusFh7xslQnAyQnQT1mNU%3D&reserved=0</div>
</blockquote>
</div>
<br class="">
</div>
</body>
</html>