[gdal-dev] Huge Memory Consumption of GDALDefaultOverviews::Initialize

[Julian Zeidler]

Hey all,

I have the Problem that my cpp-Code which exclusively uses gdal for IO 
suddenly uses vast Amounts of memory in a production environment.
When i analyzed the memory foot-print (via valgrind  massif) it turns 
out that 95% (in the test case ==7.8GB! RAM) is allocated by gdal in 
GDALDefaultOverviews::Initialize
at the line  papszInitSiblingFiles = CSLDuplicate(papszSiblingFiles);
(gdaldefaultoverviews.cpp:155, i am running gdal 1.11.2, but code is 
identical on github)

The call history is:

80.48% (8,291,659,680B) (heap allocation functions) malloc/new/new[], 
--alloc-fns, etc.
->78.78% (8,116,771,916B) 0x53C1A51: CPLMalloc (cpl_conv.cpp:136)
| ->67.21% (6,924,674,689B) 0x53C1B6A: CPLStrdup (cpl_conv.cpp:255)
| | ->67.21% (6,924,640,837B) 0x53D6BC3: CSLDuplicate (cpl_string.cpp:220)
| | | ->67.21% (6,924,640,800B) 0x538A7CA: 
GDALDefaultOverviews::Initialize(GDALDataset*, char const*, char**, int) 
(gdaldefaultoverviews.cpp:155)
| | | | ->67.21% (6,924,640,800B) 0x51C67C6: 
GTiffDataset::Open(GDALOpenInfo*) (geotiff.cpp:6437)
| | | |   ->67.21% (6,924,640,800B) 0x53887FB: 
GDALOpenInternal(GDALOpenInfo&, char const* const*) (gdaldataset.cpp:2314)
| | | |     ->67.21% (6,924,640,800B) 0x5388A24: GDALOpenInternal(char 
const*, GDALAccess, char const* const*) (gdaldataset.cpp:2263)


The CornerCases are my Programm opens ~300 geotiffs for Reading from a 
Folder with ~450.000 Files in it.

Is there any way i can have gdal not check for overviews or how i can 
free this memory? After opening a tiff file?
And should gdal be hogging this much memory while it is just checking 
for external overviews?


Cheers
Julian Zeidler