[gdal-dev] GDAL Overestimating Physical Memory

[Angus Dickey]

Even,


Thanks for the reply, I went ahead and compiled the latest GDAL 3.6.2 on
Ubuntu 22.04. Unfortunately I ended up with a similar result, GDAL thinks
it has 755GB of RAM to work with when it only has 2GB:


$ gdalinfo --version
GDAL 3.6.2, released 2023/01/02 (debug build)

$ ./get_gdal_memory
GDAL version is 3.6.2
GDAL thinks is has 811526475776 bytes of physical memory
GDAL thinks it has 811526475776 bytes of usable physical memory

$ free -h
               total        used        free      shared  buff/cache
available
Mem:           2.0Gi       148Mi       1.2Gi       0.0Ki       639Mi
1.8Gi
Swap:          256Mi          0B       256Mi

My knowledge on the subject is limited but I think Linux containers (LXC)
uses cgroups and not setrlimit to limit resources, so maybe that is why the
new changes had no effect. To reproduce this issue you can create a
container using  LXC, LXD, or a hypervision like proxmox (what I am using)
and call CPLGetUsablePhysicalRAM().

If there is any other info that might be helpful let me know. I might try a
Docker container (it also uses cgroups) and is more popular than LXC,
although it fulfills a different function.

thanks,

Angus


On Tue, Jan 24, 2023 at 5:50 PM Even Rouault <even.rouault at spatialys.com>
wrote:

> Angus,
>
> there has been a recent extra fix that landed in GDAL 3.6.2 that might
> possibly help: https://github.com/OSGeo/gdal/pull/6926
>
> Even
> Le 25/01/2023 à 01:36, Angus Dickey a écrit :
>
> Hi all,
>
> I am running into an issue where GDAL is overestimating the amount of
> physical memory it has leading to it locking up the OS by taking 100% of
> the memory. Here is an example program that illustrates the issue:
>
> #include <stdio.h>
> #include "gdal.h"
>
> int main(void) {
>    printf("GDAL version is %s\n", GDALVersionInfo("RELEASE_NAME"));
>    printf("GDAL thinks is has %lld bytes of physical memory\n",
> CPLGetPhysicalRAM());
>    printf("GDAL thinks it has %lld bytes of usable physical memory\n",
> CPLGetUsablePhysicalRAM());
>    return 0;
> }
>
> When this is compiled with GDAL 3.5.1 on Ubuntu 22.04 we get:
>
> $ ./get_gdal_memory
> GDAL version is 3.5.1
> GDAL thinks is has 811526475776 bytes of physical memory
> GDAL thinks it has 811526475776 bytes of usable physical memory
>
> Which is not consistent with the actual available memory:
>
> $ free -h
>                total        used        free      shared  buff/cache
> available
> Mem:           2.0Gi       148Mi       1.2Gi       0.0Ki       639Mi
> 1.8Gi
> Swap:          256Mi          0B       256Mi
>
> So GDAL thinks it has 755GB of memory but it only has 2GB, this causes
> issues with the raster read cache and maybe elsewhere. I suspect this is
> happening because it is running in a Linux container
> <https://linuxcontainers.org/> and GDAL is getting the total physical
> memory of the host, not the container. The strange thing is Linux
> containers use cgroups for memory restrictions and the API docs mention
> it was fixed in GDAL 2.4.0
> <https://gdal.org/api/cpl.html#_CPPv417CPLGetPhysicalRAMv> but I am still
> seeing the issue in 3.5.1.
>
> Any help or insight would be appreciated; I am happy to provide any
> additional information or testing.
>
> Thanks,
>
> Angus
>
> _______________________________________________
> gdal-dev mailing listgdal-dev at lists.osgeo.orghttps://lists.osgeo.org/mailman/listinfo/gdal-dev
>
> -- http://www.spatialys.com
> My software is free, but my time generally not.
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20230124/488d304c/attachment.htm>