[gdal-dev] Huge Memory Consumption of GDALDefaultOverviews::Initialize

[Even Rouault]

Julian,

I addressed a similar use case some time ago in GDAL 2.1 with
https://trac.osgeo.org/gdal/changeset/32242

For previous versions, you can set GDAL_DISABLE_READDIR_ON_OPEN = YES as 
environment variable / configuration option.
And if you don't need checking for any side-car file at all, you can even set
GDAL_DISABLE_READDIR_ON_OPEN = EMPTY_DIR

Even

> Hey all,
> 
> I have the Problem that my cpp-Code which exclusively uses gdal for IO
> suddenly uses vast Amounts of memory in a production environment.
> When i analyzed the memory foot-print (via valgrind  massif) it turns
> out that 95% (in the test case ==7.8GB! RAM) is allocated by gdal in
> GDALDefaultOverviews::Initialize
> at the line  papszInitSiblingFiles = CSLDuplicate(papszSiblingFiles);
> (gdaldefaultoverviews.cpp:155, i am running gdal 1.11.2, but code is
> identical on github)
> 
> The call history is:
> 
> 80.48% (8,291,659,680B) (heap allocation functions) malloc/new/new[],
> --alloc-fns, etc.
> ->78.78% (8,116,771,916B) 0x53C1A51: CPLMalloc (cpl_conv.cpp:136)
> 
> | ->67.21% (6,924,674,689B) 0x53C1B6A: CPLStrdup (cpl_conv.cpp:255)
> | 
> | | ->67.21% (6,924,640,837B) 0x53D6BC3: CSLDuplicate (cpl_string.cpp:220)
> | | 
> | | | ->67.21% (6,924,640,800B) 0x538A7CA:
> GDALDefaultOverviews::Initialize(GDALDataset*, char const*, char**, int)
> (gdaldefaultoverviews.cpp:155)
> 
> | | | | ->67.21% (6,924,640,800B) 0x51C67C6:
> GTiffDataset::Open(GDALOpenInfo*) (geotiff.cpp:6437)
> 
> | | | |   ->67.21% (6,924,640,800B) 0x53887FB:
> GDALOpenInternal(GDALOpenInfo&, char const* const*) (gdaldataset.cpp:2314)
> 
> | | | |     ->67.21% (6,924,640,800B) 0x5388A24: GDALOpenInternal(char
> 
> const*, GDALAccess, char const* const*) (gdaldataset.cpp:2263)
> 
> 
> The CornerCases are my Programm opens ~300 geotiffs for Reading from a
> Folder with ~450.000 Files in it.
> 
> Is there any way i can have gdal not check for overviews or how i can
> free this memory? After opening a tiff file?
> And should gdal be hogging this much memory while it is just checking
> for external overviews?
> 
> 
> Cheers
> Julian Zeidler
> _______________________________________________
> gdal-dev mailing list
> gdal-dev at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/gdal-dev

-- 
Spatialys - Geospatial professional services
http://www.spatialys.com