[gdal-dev] Call for discussion on RFC 26: GDAL Block Cache Improvements

[Even Rouault]

I'll call soon on a vote on this if nobody has remarks.

Le jeudi 04 juin 2015 16:05:00, Even Rouault a écrit :
> Hi,
> 
> I've updated an old RFC initiated by Tamas. The main idea, having a hashset
> based implementation as an alternative to the array based, remains. Changes
> consist mainly in code restructuration, perf improvements to reduce lock
> contention and porting to the state of the latest code base.
> 
> This is a RFC for GDAL 2.1
> 
> Details at https://trac.osgeo.org/gdal/wiki/rfc26_blockcache
> 
> == Summary and rationale ==
> 
> GDAL maintains an in-memory cache for the raster blocks fetched from the
> drivers and ensures that the second attempt to access the same block will
> be served from the cache instead of the driver. This cache is maintained
> in a per-band fashion and an array is allocated for the pointers for each
> blocks (or sub-blocks). This approach is not sufficient with large raster
> dimensions (or large virtual rasters ie. with the WMS/TMS driver), which
> may cause out of memory errors in GDALRasterBand::InitBlockInfo, as raised
> in #3224
> 
> For example, a band of a dataset at level 21 with a GoogleMaps tiling
> requires 2097152x2097152 tiles of 256x256 pixels. This means that GDAL
> will try to allocate an array of 32768x32768 = 1 billion elements (32768 =
> 2097152 / 64). The size of this array is 4 GB on a 32-bit build, so it
> cannot be allocated at all. And it is 8 GB on a 64-bit build (even if this
> is generally only virtual memory reservation but not actually allocation
> of physical pages of memory, due to over-commit mechanism of the operating
> system). At dataset closing, this means that those 1 billion cells will
> have to be explored to discover remaining cached blocks. In reality, all
> above figures must be multiplied by 3 for a RGB (or 4 for a RGBA) dataset.
> 
> In the hash set implementation, memory allocation depends directly on the
> number of cached blocks. Typically with the default GDAL_CACHEMAX size of
> 40 MB, only 640 blocks of 256x256 pixels can be simultaneously cached (for
> all datasets).
> 
> Best regards,
> 
> Even

-- 
Spatialys - Geospatial professional services
http://www.spatialys.com