[gdal-dev] Huge Memory Consumption of GDALDefaultOverviews::Initialize

[Even Rouault]

Le mercredi 06 avril 2016 15:21:24, Julian Zeidler a écrit :
> Hi Even,
> 
> Thank you for the the very quick and very helpful reply. I spent the
> whole day yesterday trying to find the memory leak in my code that
> caused 11GB Ram usage. ;-)
> One quick follow-up would setting GDAL_DISABLE_READDIR_ON_OPEN =
> EMPTY_DIR break reading from vrt or envi files with a header?

Should work with VRT (VRT sources are not really side-car files in the meaning 
I meant), but that would break ENVI.
I'd recommend just using GDAL_DISABLE_READDIR_ON_OPEN = YES to be safe. That 
way drivers that need sidecar files will still try to probe them (directly, and 
not by doing a readdir()).

> 
> Cheers
> Julian
> 
> Am 06.04.2016 um 15:09 schrieb Even Rouault:
> > Julian,
> > 
> > I addressed a similar use case some time ago in GDAL 2.1 with
> > https://trac.osgeo.org/gdal/changeset/32242
> > 
> > For previous versions, you can set GDAL_DISABLE_READDIR_ON_OPEN = YES as
> > environment variable / configuration option.
> > And if you don't need checking for any side-car file at all, you can even
> > set GDAL_DISABLE_READDIR_ON_OPEN = EMPTY_DIR
> > 
> > Even
> > 
> >> Hey all,
> >> 
> >> I have the Problem that my cpp-Code which exclusively uses gdal for IO
> >> suddenly uses vast Amounts of memory in a production environment.
> >> When i analyzed the memory foot-print (via valgrind  massif) it turns
> >> out that 95% (in the test case ==7.8GB! RAM) is allocated by gdal in
> >> GDALDefaultOverviews::Initialize
> >> at the line  papszInitSiblingFiles = CSLDuplicate(papszSiblingFiles);
> >> (gdaldefaultoverviews.cpp:155, i am running gdal 1.11.2, but code is
> >> identical on github)
> >> 
> >> The call history is:
> >> 
> >> 80.48% (8,291,659,680B) (heap allocation functions) malloc/new/new[],
> >> --alloc-fns, etc.
> >> ->78.78% (8,116,771,916B) 0x53C1A51: CPLMalloc (cpl_conv.cpp:136)
> >> 
> >> | ->67.21% (6,924,674,689B) 0x53C1B6A: CPLStrdup (cpl_conv.cpp:255)
> >> | 
> >> | | ->67.21% (6,924,640,837B) 0x53D6BC3: CSLDuplicate
> >> | | (cpl_string.cpp:220)
> >> | | 
> >> | | | ->67.21% (6,924,640,800B) 0x538A7CA:
> >> GDALDefaultOverviews::Initialize(GDALDataset*, char const*, char**, int)
> >> (gdaldefaultoverviews.cpp:155)
> >> 
> >> | | | | ->67.21% (6,924,640,800B) 0x51C67C6:
> >> GTiffDataset::Open(GDALOpenInfo*) (geotiff.cpp:6437)
> >> 
> >> | | | |   ->67.21% (6,924,640,800B) 0x53887FB:
> >> GDALOpenInternal(GDALOpenInfo&, char const* const*)
> >> (gdaldataset.cpp:2314)
> >> 
> >> | | | |     ->67.21% (6,924,640,800B) 0x5388A24: GDALOpenInternal(char
> >> 
> >> const*, GDALAccess, char const* const*) (gdaldataset.cpp:2263)
> >> 
> >> 
> >> The CornerCases are my Programm opens ~300 geotiffs for Reading from a
> >> Folder with ~450.000 Files in it.
> >> 
> >> Is there any way i can have gdal not check for overviews or how i can
> >> free this memory? After opening a tiff file?
> >> And should gdal be hogging this much memory while it is just checking
> >> for external overviews?
> >> 
> >> 
> >> Cheers
> >> Julian Zeidler
> >> _______________________________________________
> >> gdal-dev mailing list
> >> gdal-dev at lists.osgeo.org
> >> http://lists.osgeo.org/mailman/listinfo/gdal-dev

-- 
Spatialys - Geospatial professional services
http://www.spatialys.com