[Gdal-dev] Level 2 caching mechanism question

[Frank Warmerdam]

Daniele Romagnoli wrote:
> Hi guys.
> I'm writing a brief report on some GDAL capabilities and I would want to 
> know if I have correctly understood the level 2 caching mechanism.
> 
> Anyone can briefly summarize to me, how such a mechanism works as well 
> as its advantages?
> 
> Maybe, I have found the answer by analyzing GDAL sourcecode (IRasterIO 
> method and Co.).  (As an instance, If there are more than 64 blocks in a 
> row, then the sub blocking is activated by grouping 64 blocks at once in 
> greater subblocks...)
> 
> But I would like to have a better vision of the mechanism prior to write 
> wrong things.
> Anyone could help me please?

Daniele,

Conceptually GDAL has a single level block cache.  However, for large files
with small blocks there can be a lot of blocks.  So many that maintaining
a single pointer to each block can get quite expensive in memory.

To avoid that the caching code maintains a two level data structure when
there are large numbers of blocks - well really any time the image is more
than 64 blocks wide.  The first level of the datastructure keeps a pointer
to each 64x64 subgroup of block pointers with the high level pointer being
NULL if the subarea has never been touched allowing 4096 blocks to be
represented by a single pointer if the region is not processed.

This two level data structure was implemented mainly to support Erdas
Imagine files in the 100+ GB range efficiently.  Imagine files typically
have fairly small blocks (64pixels x 64lines) so big files can have a lot
of blocks.

But I want to stress that this is not a "two level cache" in the sense
that computer processors have two levels of memory.  That is to say there
isn't a "fast cache" and "slow cache".  Just a two level data structure to
track a one level cache.

Best regards,
-- 
---------------------------------------+--------------------------------------
I set the clouds in motion - turn up   | Frank Warmerdam, warmerdam at pobox.com
light and sound - activate the windows | http://pobox.com/~warmerdam
and watch the world go round - Rush    | President OSGF, http://osgeo.org