[gdal-dev] GDAL Overestimating Physical Memory

[Angus Dickey]

Hi all,

I am running into an issue where GDAL is overestimating the amount of
physical memory it has leading to it locking up the OS by taking 100% of
the memory. Here is an example program that illustrates the issue:

#include <stdio.h>
#include "gdal.h"

int main(void) {
   printf("GDAL version is %s\n", GDALVersionInfo("RELEASE_NAME"));
   printf("GDAL thinks is has %lld bytes of physical memory\n",
CPLGetPhysicalRAM());
   printf("GDAL thinks it has %lld bytes of usable physical memory\n",
CPLGetUsablePhysicalRAM());
   return 0;
}

When this is compiled with GDAL 3.5.1 on Ubuntu 22.04 we get:

$ ./get_gdal_memory
GDAL version is 3.5.1
GDAL thinks is has 811526475776 bytes of physical memory
GDAL thinks it has 811526475776 bytes of usable physical memory

Which is not consistent with the actual available memory:

$ free -h
               total        used        free      shared  buff/cache
available
Mem:           2.0Gi       148Mi       1.2Gi       0.0Ki       639Mi
1.8Gi
Swap:          256Mi          0B       256Mi

So GDAL thinks it has 755GB of memory but it only has 2GB, this causes
issues with the raster read cache and maybe elsewhere. I suspect this is
happening because it is running in a Linux container
<https://linuxcontainers.org/> and GDAL is getting the total physical
memory of the host, not the container. The strange thing is Linux
containers use cgroups for memory restrictions and the API docs mention it
was fixed in GDAL 2.4.0
<https://gdal.org/api/cpl.html#_CPPv417CPLGetPhysicalRAMv> but I am still
seeing the issue in 3.5.1.

Any help or insight would be appreciated; I am happy to provide any
additional information or testing.

Thanks,

Angus
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20230124/b9a2b6ff/attachment.htm>