[Gdal-dev] Two Level Raster Caches

[Ed McNierney]

Frank -

Thanks - this is a nice solution.  Several years ago I implemented a
GDAL driver for Microsoft TerraServer, which I finished just about the
time they added WMS support <g>.  I had a similar problem, since this
(virtual) dataset had a HUGE number of relatively small tiles.  It took
me a while to figure out why my MapServer working set was several
hundred megabytes.

In that case I implemented a linked-list cache to replace the array
cache.  This was a good solution for MapServer's use of GDAL, since any
given run would load a very small number of tiles and typically use each
one only a very few times.

I never contributed the code because I was concerned about having two
different implementations of the cache without a good way of having the
client specify which one was to be used.  In many applications the
linked-list cache would be a very poor choice.  A two-level cache is a
much better "middle ground" that should work well as an all-around
solution.

	- Ed

Ed McNierney
TopoZone.com

-----Original Message-----
From: gdal-dev-bounces at xserve.flids.com
[mailto:gdal-dev-bounces at xserve.flids.com] On Behalf Of Frank Warmerdam
Sent: Thursday, February 17, 2005 5:46 PM
To: gdal-dev at remotesensing.org
Subject: [Gdal-dev] Two Level Raster Caches

Folks,

Traditionally GDALRasterBands maintain a block cache.  Internally 
this is an array of pointers to GDALRasterBlock objects.  If a block
hasn't
been read, the pointer is NULL. 

However, for very large images with relatively small tile sizes it is
possible to have *alot* of block pointers to store.  For instance, a 
1 million by 1 million image with 64x64 tiles would be 244 million
blocks.  The block pointer array alone for this image would be 1GB.

While this is a pretty big image, I have had real-world performance
issues due to the size of the cache pointer array.   

With Peng's encouragement, I have changed to a two-level cache
in some cases.  Basically the first level cache becomes an array 
of pointers to secondary grids which are 64x64 blocks. 

I have done *some* testing of this change, and committed it today.  It
shouldn't cause any problems, but if folks have an issue please let me
know. 

I would add the two level caching only kicks in for images at least
32 blocks wide. 

Best regards,
-- 
---------------------------------------+--------------------------------
------
I set the clouds in motion - turn up   | Frank Warmerdam,
warmerdam at pobox.com
light and sound - activate the windows | http://pobox.com/~warmerdam
and watch the world go round - Rush    | Geospatial Programmer for Rent
_______________________________________________
Gdal-dev mailing list
Gdal-dev at xserve.flids.com
http://xserve.flids.com/mailman/listinfo/gdal-dev