<!DOCTYPE html><html><head><title></title><style type="text/css">p.MsoNormal,p.MsoNoSpacing{margin:0}</style></head><body><div style="font-family:Arial;">Hello,<br></div><div style="font-family:Arial;"><br></div><div style="font-family:Arial;">I also managed to reproduce this in Docker (4 GB limit):<br></div><div style="font-family:Arial;"><br></div><div style="font-family:Arial;"><span class="font" style="font-family:menlo, consolas, monospace, sans-serif;"># cat /proc/self/cgroup </span><span class="font" style="font-family:menlo, consolas, monospace, sans-serif;"><br></span></div><div style="font-family:Arial;"><span class="font" style="font-family:menlo, consolas, monospace, sans-serif;">0::/</span></div><div style="font-family:Arial;"><span class="font" style="font-family:menlo, consolas, monospace, sans-serif;"># cat /sys/fs/cgroup/memory.max</span><span class="font" style="font-family:menlo, consolas, monospace, sans-serif;"><br></span></div><div style="font-family:Arial;"><span class="font" style="font-family:menlo, consolas, monospace, sans-serif;">4294967296</span><span class="font" style="font-family:menlo, consolas, monospace, sans-serif;"><br></span></div><div style="font-family:Arial;"><span class="font" style="font-family:menlo, consolas, monospace, sans-serif;"># autotest/cpp/gdallimits </span><span class="font" style="font-family:menlo, consolas, monospace, sans-serif;"><br></span></div><div style="font-family:Arial;"><span class="font" style="font-family:menlo, consolas, monospace, sans-serif;">CPLGetNumCPUs = 32</span><span class="font" style="font-family:menlo, consolas, monospace, sans-serif;"><br></span></div><div style="font-family:Arial;"><span class="font" style="font-family:menlo, consolas, monospace, sans-serif;">CPLGetUsablePhysicalRAM = 62 GB</span><span class="font" style="font-family:menlo, consolas, monospace, sans-serif;"><br></span></div><div style="font-family:Arial;"><br></div><div style="font-family:Arial;">(<span class="font" style="font-family:menlo, consolas, monospace, sans-serif;">podman</span> behaves exactly the same)<br></div><div style="font-family:Arial;"><br></div><div style="font-family:Arial;">Laurentiu<br></div><div style="font-family:Arial;"><br></div><div>On Thu, Jan 26, 2023, at 03:13, Angus Dickey wrote:<br></div><blockquote type="cite" id="qt" style=""><div dir="ltr"><div>Even,<br></div><div><br></div><div>Thanks, that is some quick turn around! I imagine <a href="https://www.proxmox.com/en/">Proxmox</a> or <a href="https://linuxcontainers.org/lxd/introduction/">LXD</a> are pretty much what everyone uses to create linux containers. LXC is the underlying technology but also has a set of command line tools that can be used to create containers. In your case it sounds like LXD can't choose a subnet for your linux bridge, which is mysterious and I don't know how to fix that.<br></div><div><br></div><div>I tried your update inside a container and am still seeing the problem where GDAL thinks it has the full host memory:<br></div><div><div><br></div><div><span class="font" style="font-family:monospace;">$ gdalinfo --version<br>GDAL 3.7.0dev, released 2023/99/99 (debug build)<br>$ ./get_gdal_memory</span></div></div><div><span class="font" style="font-family:monospace;">GDAL version is 3.7.0dev<br>GDAL thinks it has 135083474944 bytes of physical memory<br>GDAL thinks it has 135083474944 bytes of usable physical memory<br>sysinfo() thinks it has 135083474944 bytes of physical memory</span></div><div><span class="font" style="font-family:monospace;">$ free -h<br> total used free shared buff/cache available<br>Mem: 2.0Gi 152Mi 1.1Gi 0.0Ki 755Mi 1.8Gi<br>Swap: 256Mi 0B 256Mi</span></div><div><span class="font" style="font-family:monospace;">$ cat /proc/meminfo | grep MemTotal<br>MemTotal: 2048000 kB</span></div><div><div><br></div><div>I wanted to dig a bit but am no expert in containerization and cgroup v2. It seems that some tools show the memory the container has (<span class="font" style="font-family:monospace;"><a href="https://man7.org/linux/man-pages/man1/free.1.html">free </a></span>& <span class="font" style="font-family:monospace;"><a href="https://man7.org/linux/man-pages/man5/proc.5.html">/proc/meminfo</a></span>) and others (<span style="font-family:monospace;"><span class="font" style="font-family:monospace;"><a href="https://man7.org/linux/man-pages/man2/sysinfo.2.html">sysinfo</a></span></span>) show the host memory. For cgroups v2 I see your code is trying to find the max memory from a specific <span class="font" style="font-family:monospace;">memory.max</span> file in <span class="font" style="font-family:monospace;">/sys/fs/cgroup/</span><span class="font" style="font-family:arial, sans-serif;">. In my <i>containers </i>that file (actually all the </span><span class="font" style="font-family:monospace;">memory.max</span><span class="font" style="font-family:arial, sans-serif;"> files) contain the default value "max".</span><br></div><div><span class="font" style="font-family:arial, sans-serif;"></span><br></div><div><span class="font" style="font-family:monospace;">$ find /sys/fs/cgroup -type f -name memory.max -exec sh -c "cat '{}'" \;<br>max<br>max<br>max<br>... all max ...<br>max</span></div><div><span class="font" style="font-family:monospace;"></span><br></div><div><span class="font" style="font-family:arial, sans-serif;">If I try the same thing on the <i>host </i>I actually find it is set to the expected value.</span><br></div><div><span class="font" style="font-family:arial, sans-serif;"></span><br></div><div><span class="font" style="font-family:monospace;">cat $ /sys/fs/cgroup/lxc/901/memory.max<br>2097152000</span><span class="font" style="font-family:arial, sans-serif;"></span></div><div><span class="font" style="font-family:monospace;"></span><br></div><div><span class="font" style="font-family:arial, sans-serif;">The cgroup values on the host appear to be what is limiting the container memory, more rules can be added inside the container but they are still beholden to the host rules. I am not sure how </span><span class="font" style="font-family:monospace;">free </span>& <span class="font" style="font-family:monospace;">/proc/memory</span><span class="font" style="font-family:arial, sans-serif;"> are getting the correct available memory but maybe I will ask the proxmox or LXD people.</span><br></div><div><span class="font" style="font-family:arial, sans-serif;"></span><br></div><div><span class="font" style="font-family:arial, sans-serif;">Thanks again,</span><br></div><div><span class="font" style="font-family:arial, sans-serif;"></span><br></div><div><span class="font" style="font-family:arial, sans-serif;">Angus</span><br></div><div><br></div></div></div><div><br></div><div class="qt-gmail_quote"><div dir="ltr" class="qt-gmail_attr">On Wed, Jan 25, 2023 at 4:49 AM Even Rouault <<a href="mailto:even.rouault@spatialys.com">even.rouault@spatialys.com</a>> wrote:<br></div><blockquote class="qt-gmail_quote" style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0.8ex;border-left-color:rgb(204, 204, 204);border-left-style:solid;border-left-width:1px;padding-left:1ex;"><div><p>Angus,<br></p><p>I'm not familiar with LXC. I tried to setup LXD with <a href="https://linuxcontainers.org/lxd/introduction/" target="_blank">https://linuxcontainers.org/lxd/introduction/</a> but it fails with a
mysterious "Error: Failed to create local member network "lxdbr0"
in project "default": Failed generating auto config: Failed to
automatically find an unused IPv4 subnet, manual configuration
required"<br></p><p>Anyway, I've attempted in <a href="https://github.com/OSGeo/gdal/pull/7124" target="_blank">https://github.com/OSGeo/gdal/pull/7124</a> to better take into account cgroup to get memory limitation. Could
you give this a try?<br></p><p>Even<br></p><div>Le 25/01/2023 à 06:24, Angus Dickey a
écrit :<br></div><blockquote type="cite"><div dir="ltr"><div dir="ltr"><div><div dir="ltr"><div dir="ltr"><span><p style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;">Even,<br></p><p style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><br></p><p style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;">Thanks
for the reply, I went ahead and compiled the latest
GDAL 3.6.2 on Ubuntu 22.04. Unfortunately I ended up
with a similar result, GDAL thinks it has 755GB of
RAM to work with when it only has 2GB:<br></p><p style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><br></p><p style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span class="font" style="font-family:monospace;">$ gdalinfo --version<br> GDAL 3.6.2, released 2023/01/02 (debug build)<br> <br> $ ./get_gdal_memory<br> GDAL version is 3.6.2<br> GDAL thinks is has 811526475776 bytes of physical
memory<br> GDAL thinks it has 811526475776 bytes of usable
physical memory<br> <br> $ free -h<br> total used free
shared buff/cache available<br> Mem: 2.0Gi 148Mi 1.2Gi
0.0Ki 639Mi 1.8Gi<br> Swap: 256Mi 0B 256Mi</span></p></span></div></div></div></div><div><br></div><div>My knowledge on the subject is limited but I think Linux
containers (LXC) uses cgroups and not setrlimit to limit
resources, so maybe that is why the new changes had no effect.
To reproduce this issue you can create a container using LXC,
LXD, or a hypervision like proxmox (what I am using) and call
CPLGetUsablePhysicalRAM().<br></div><div><br></div><div><div>If there is any other info that might be helpful let me
know. I might try a Docker container (it also uses cgroups)
and is more popular than LXC, although it fulfills a different
function.<br></div><div><br></div><div>thanks,<br></div><div><br></div><div><div>Angus<br></div><div><br></div><div><div><br></div><div class="qt-gmail_quote"><div dir="ltr" class="qt-gmail_attr">On Tue, Jan 24, 2023
at 5:50 PM Even Rouault <<a href="mailto:even.rouault@spatialys.com" target="_blank">even.rouault@spatialys.com</a>>
wrote:<br></div><blockquote class="qt-gmail_quote" style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0.8ex;border-left-color:rgb(204, 204, 204);border-left-style:solid;border-left-width:1px;padding-left:1ex;"><div><p>Angus,<br></p><p>there has been a recent extra fix that landed in
GDAL 3.6.2 that might possibly help: <a href="https://github.com/OSGeo/gdal/pull/6926" target="_blank">https://github.com/OSGeo/gdal/pull/6926</a><br></p><p>Even<br></p><div>Le 25/01/2023 à 01:36, Angus Dickey a écrit :<br></div><blockquote type="cite"><div dir="ltr"><div>Hi all,<br></div><div><br></div><div><div>I am running into an issue where GDAL is
overestimating the amount of physical
memory it has leading to it locking up the OS
by taking 100% of the memory. Here is an
example program that illustrates the issue:<br></div><div> <br></div><div> #include <stdio.h><br></div><div> #include "gdal.h"<br></div><div> <br></div><div> int main(void) {<br></div><div> printf("GDAL version is %s\n",
GDALVersionInfo("RELEASE_NAME"));<br></div><div> printf("GDAL thinks is has %lld bytes of
physical memory\n", CPLGetPhysicalRAM());<br></div><div> printf("GDAL thinks it has %lld bytes of
usable physical memory\n",
CPLGetUsablePhysicalRAM());<br></div><div> return 0;<br></div><div> }<br></div></div><div><br></div><div>When this is compiled with GDAL 3.5.1 on
Ubuntu 22.04 we get:<br></div><div><br></div><div><div>$ ./get_gdal_memory <br></div><div> GDAL version is 3.5.1<br></div><div> GDAL thinks is has 811526475776 bytes of
physical memory<br></div><div> GDAL thinks it has 811526475776 bytes of
usable physical memory<br></div><div> <br></div><div> Which is not consistent with the actual
available memory:<br></div></div><div><div><br></div><div>$ free -h<br></div><div> total used free
shared buff/cache available<br></div><div> Mem: 2.0Gi 148Mi 1.2Gi
0.0Ki 639Mi 1.8Gi<br></div><div> Swap: 256Mi 0B 256Mi<br></div></div><div><br></div><div>So GDAL thinks it has 755GB of memory but
it only has 2GB, this causes issues with the
raster read cache and maybe elsewhere. I
suspect this is happening because it is
running in a <a href="https://linuxcontainers.org/" target="_blank">Linux
container</a> and GDAL is getting the total
physical memory of the host, not the
container. The strange thing is Linux
containers use cgroups for memory restrictions
and the API docs <a href="https://gdal.org/api/cpl.html#_CPPv417CPLGetPhysicalRAMv" target="_blank">mention
it was fixed in GDAL 2.4.0</a> but I am
still seeing the issue in 3.5.1.<br></div><div><br></div><div>Any help or insight would be appreciated; I
am happy to provide any
additional information or testing.<br></div><div><br></div><div>Thanks,<br></div><div><br></div><div>Angus<br></div></div><div><br></div><pre>_______________________________________________
gdal-dev mailing list
<a href="mailto:gdal-dev@lists.osgeo.org" target="_blank">gdal-dev@lists.osgeo.org</a>
<a href="https://lists.osgeo.org/mailman/listinfo/gdal-dev" target="_blank">https://lists.osgeo.org/mailman/listinfo/gdal-dev</a>
<br></pre></blockquote><pre cols="72">--
<a href="http://www.spatialys.com" target="_blank">http://www.spatialys.com</a>
My software is free, but my time generally not.<br></pre></div></blockquote></div></div></div></div></div></blockquote><pre cols="72">--
<a href="http://www.spatialys.com" target="_blank">http://www.spatialys.com</a>
My software is free, but my time generally not.<br></pre></div></blockquote></div><div>_______________________________________________<br></div><div>gdal-dev mailing list<br></div><div><a href="mailto:gdal-dev@lists.osgeo.org">gdal-dev@lists.osgeo.org</a><br></div><div><a href="https://lists.osgeo.org/mailman/listinfo/gdal-dev">https://lists.osgeo.org/mailman/listinfo/gdal-dev</a><br></div><div><br></div></blockquote><div style="font-family:Arial;"><br></div></body></html>