<div dir="ltr"><div dir="ltr"><div><div dir="ltr" class="gmail_signature"><div dir="ltr"><span><p style="line-height:1.38;margin-top:0pt;margin-bottom:0pt">Even,</p><p style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><br></p><p style="line-height:1.38;margin-top:0pt;margin-bottom:0pt">Thanks for the reply, I went ahead and compiled the latest GDAL 3.6.2 on Ubuntu 22.04. Unfortunately I ended up with a similar result, GDAL thinks it has 755GB of RAM to work with when it only has 2GB:</p><p style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><br></p><p style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><font face="monospace">$ gdalinfo --version<br>GDAL 3.6.2, released 2023/01/02 (debug build)<br><br>$ ./get_gdal_memory<br>GDAL version is 3.6.2<br>GDAL thinks is has 811526475776 bytes of physical memory<br>GDAL thinks it has 811526475776 bytes of usable physical memory<br><br>$ free -h<br> total used free shared buff/cache available<br>Mem: 2.0Gi 148Mi 1.2Gi 0.0Ki 639Mi 1.8Gi<br>Swap: 256Mi 0B 256Mi<br></font></p></span></div></div></div></div><div><br></div>My knowledge on the subject is limited but I think Linux containers (LXC) uses cgroups and not setrlimit to limit resources, so maybe that is why the new changes had no effect. To reproduce this issue you can create a container using LXC, LXD, or a hypervision like proxmox (what I am using) and call CPLGetUsablePhysicalRAM().<div><br></div><div>If there is any other info that might be helpful let me know. I might try a Docker container (it also uses cgroups) and is more popular than LXC, although it fulfills a different function.<div><br></div><div>thanks,</div><div><br></div><div>Angus<br><div><br></div><div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Jan 24, 2023 at 5:50 PM Even Rouault <<a href="mailto:even.rouault@spatialys.com">even.rouault@spatialys.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<p>Angus,</p>
<p>there has been a recent extra fix that landed in GDAL 3.6.2 that
might possibly help: <a href="https://github.com/OSGeo/gdal/pull/6926" target="_blank">https://github.com/OSGeo/gdal/pull/6926</a></p>
<p>Even</p>
<div>Le 25/01/2023 à 01:36, Angus Dickey a
écrit :<br>
</div>
<blockquote type="cite">
<div dir="ltr">Hi all,
<div><br>
</div>
<div>I am running into an issue where GDAL is overestimating the
amount of physical memory it has leading to it locking up the
OS by taking 100% of the memory. Here is an example program
that illustrates the issue:<br>
<br>
#include <stdio.h><br>
#include "gdal.h"<br>
<br>
int main(void) {<br>
printf("GDAL version is %s\n",
GDALVersionInfo("RELEASE_NAME"));<br>
printf("GDAL thinks is has %lld bytes of physical
memory\n", CPLGetPhysicalRAM());<br>
printf("GDAL thinks it has %lld bytes of usable physical
memory\n", CPLGetUsablePhysicalRAM());<br>
return 0;<br>
}<br>
</div>
<div><br>
</div>
<div>When this is compiled with GDAL 3.5.1 on Ubuntu 22.04 we
get:<br>
</div>
<div><br>
</div>
<div>$ ./get_gdal_memory <br>
GDAL version is 3.5.1<br>
GDAL thinks is has 811526475776 bytes of physical memory<br>
GDAL thinks it has 811526475776 bytes of usable physical
memory<br>
<br>
Which is not consistent with the actual available memory:</div>
<div><br>
$ free -h<br>
total used free shared
buff/cache available<br>
Mem: 2.0Gi 148Mi 1.2Gi 0.0Ki
639Mi 1.8Gi<br>
Swap: 256Mi 0B 256Mi<br>
</div>
<div><br>
</div>
<div>So GDAL thinks it has 755GB of memory but it only has 2GB,
this causes issues with the raster read cache and maybe
elsewhere. I suspect this is happening because it is running
in a <a href="https://linuxcontainers.org/" target="_blank">Linux container</a> and GDAL is
getting the total physical memory of the host, not the
container. The strange thing is Linux containers use cgroups
for memory restrictions and the API docs <a href="https://gdal.org/api/cpl.html#_CPPv417CPLGetPhysicalRAMv" target="_blank">mention it was fixed in GDAL 2.4.0</a> but
I am still seeing the issue in 3.5.1.</div>
<div><br>
</div>
<div>Any help or insight would be appreciated; I am happy to
provide any additional information or testing.</div>
<div><br>
</div>
<div>Thanks,</div>
<div><br>
</div>
<div>Angus</div>
</div>
<br>
<fieldset></fieldset>
<pre>_______________________________________________
gdal-dev mailing list
<a href="mailto:gdal-dev@lists.osgeo.org" target="_blank">gdal-dev@lists.osgeo.org</a>
<a href="https://lists.osgeo.org/mailman/listinfo/gdal-dev" target="_blank">https://lists.osgeo.org/mailman/listinfo/gdal-dev</a>
</pre>
</blockquote>
<pre cols="72">--
<a href="http://www.spatialys.com" target="_blank">http://www.spatialys.com</a>
My software is free, but my time generally not.</pre>
</div>
</blockquote></div></div></div></div></div>