[gdal-dev] GDAL Overestimating Physical Memory

[Even Rouault]

Angus,

there has been a recent extra fix that landed in GDAL 3.6.2 that might 
possibly help: https://github.com/OSGeo/gdal/pull/6926

Even

Le 25/01/2023 à 01:36, Angus Dickey a écrit :
> Hi all,
>
> I am running into an issue where GDAL is overestimating the amount of 
> physical memory it has leading to it locking up the OS by taking 100% 
> of the memory. Here is an example program that illustrates the issue:
>
> #include <stdio.h>
> #include "gdal.h"
>
> int main(void) {
>    printf("GDAL version is %s\n", GDALVersionInfo("RELEASE_NAME"));
>    printf("GDAL thinks is has %lld bytes of physical memory\n", 
> CPLGetPhysicalRAM());
>    printf("GDAL thinks it has %lld bytes of usable physical memory\n", 
> CPLGetUsablePhysicalRAM());
>    return 0;
> }
>
> When this is compiled with GDAL 3.5.1 on Ubuntu 22.04 we get:
>
> $ ./get_gdal_memory
> GDAL version is 3.5.1
> GDAL thinks is has 811526475776 bytes of physical memory
> GDAL thinks it has 811526475776 bytes of usable physical memory
>
> Which is not consistent with the actual available memory:
>
> $ free -h
>                total        used        free      shared  buff/cache   
> available
> Mem:           2.0Gi       148Mi       1.2Gi       0.0Ki 639Mi       1.8Gi
> Swap:          256Mi          0B       256Mi
>
> So GDAL thinks it has 755GB of memory but it only has 2GB, this causes 
> issues with the raster read cache and maybe elsewhere. I suspect this 
> is happening because it is running in a Linux container 
> <https://linuxcontainers.org/> and GDAL is getting the total physical 
> memory of the host, not the container. The strange thing is Linux 
> containers use cgroups for memory restrictions and the API docs 
> mention it was fixed in GDAL 2.4.0 
> <https://gdal.org/api/cpl.html#_CPPv417CPLGetPhysicalRAMv> but I am 
> still seeing the issue in 3.5.1.
>
> Any help or insight would be appreciated; I am happy to provide any 
> additional information or testing.
>
> Thanks,
>
> Angus
>
> _______________________________________________
> gdal-dev mailing list
> gdal-dev at lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/gdal-dev

-- 
http://www.spatialys.com
My software is free, but my time generally not.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20230125/4fff862f/attachment.htm>