[gdal-dev] Huge Memory Consumption of GDALDefaultOverviews::Initialize

[Julian Zeidler]

Hi Even,

Thank you for the the very quick and very helpful reply. I spent the 
whole day yesterday trying to find the memory leak in my code that 
caused 11GB Ram usage. ;-)
One quick follow-up would setting GDAL_DISABLE_READDIR_ON_OPEN = 
EMPTY_DIR break reading from vrt or envi files with a header?

Cheers
Julian

Am 06.04.2016 um 15:09 schrieb Even Rouault:
> Julian,
>
> I addressed a similar use case some time ago in GDAL 2.1 with
> https://trac.osgeo.org/gdal/changeset/32242
>
> For previous versions, you can set GDAL_DISABLE_READDIR_ON_OPEN = YES as
> environment variable / configuration option.
> And if you don't need checking for any side-car file at all, you can even set
> GDAL_DISABLE_READDIR_ON_OPEN = EMPTY_DIR
>
> Even
>
>> Hey all,
>>
>> I have the Problem that my cpp-Code which exclusively uses gdal for IO
>> suddenly uses vast Amounts of memory in a production environment.
>> When i analyzed the memory foot-print (via valgrind  massif) it turns
>> out that 95% (in the test case ==7.8GB! RAM) is allocated by gdal in
>> GDALDefaultOverviews::Initialize
>> at the line  papszInitSiblingFiles = CSLDuplicate(papszSiblingFiles);
>> (gdaldefaultoverviews.cpp:155, i am running gdal 1.11.2, but code is
>> identical on github)
>>
>> The call history is:
>>
>> 80.48% (8,291,659,680B) (heap allocation functions) malloc/new/new[],
>> --alloc-fns, etc.
>> ->78.78% (8,116,771,916B) 0x53C1A51: CPLMalloc (cpl_conv.cpp:136)
>>
>> | ->67.21% (6,924,674,689B) 0x53C1B6A: CPLStrdup (cpl_conv.cpp:255)
>> |
>> | | ->67.21% (6,924,640,837B) 0x53D6BC3: CSLDuplicate (cpl_string.cpp:220)
>> | |
>> | | | ->67.21% (6,924,640,800B) 0x538A7CA:
>> GDALDefaultOverviews::Initialize(GDALDataset*, char const*, char**, int)
>> (gdaldefaultoverviews.cpp:155)
>>
>> | | | | ->67.21% (6,924,640,800B) 0x51C67C6:
>> GTiffDataset::Open(GDALOpenInfo*) (geotiff.cpp:6437)
>>
>> | | | |   ->67.21% (6,924,640,800B) 0x53887FB:
>> GDALOpenInternal(GDALOpenInfo&, char const* const*) (gdaldataset.cpp:2314)
>>
>> | | | |     ->67.21% (6,924,640,800B) 0x5388A24: GDALOpenInternal(char
>>
>> const*, GDALAccess, char const* const*) (gdaldataset.cpp:2263)
>>
>>
>> The CornerCases are my Programm opens ~300 geotiffs for Reading from a
>> Folder with ~450.000 Files in it.
>>
>> Is there any way i can have gdal not check for overviews or how i can
>> free this memory? After opening a tiff file?
>> And should gdal be hogging this much memory while it is just checking
>> for external overviews?
>>
>>
>> Cheers
>> Julian Zeidler
>> _______________________________________________
>> gdal-dev mailing list
>> gdal-dev at lists.osgeo.org
>> http://lists.osgeo.org/mailman/listinfo/gdal-dev