Thoughts on a feature cache for queries...

[Frank Warmerdam]

Steve Lime wrote:
> Hi all: What would folks think about implementing a simple feature cache for
> storing query results? I know Tamas has something grander planned but it
> seems a ways off and implementation with queries may be problematic anyway.
> We could use the featureList structures that form a linked list containing
> shapeObj's. I've always been a bit hesitant to do this because with large
> datasets you could use a ton of memory, but the performance bottlenecks with
> the two-pass query probably out weigh that problem. If folks are somewhat
> positive about the idea I'd be glad to work up an RFC...

Steve,

I think this is an important development and worth an RFC to work out the
details.

I'm hesitant about using the featureList linked list approach, as that is
hard to access quickly except by linear traversal.

I am imagining some sort of feature cache in the layerObj which
msLayerGetShape() would first check before calling the provider specific
LayerGetShape() function.  The cache could use a hash table approach or
perhaps an ordered array allowing binary searches.

Then we could choose whether or not to populate this cache with query
results based on the provider type or perhaps some flag in the layer object
from the map file.  That way the mechanism wouldn't be used for layer types,
such as shapefiles, that allow fast by-index lookups anyways.

I am kicking myself for doing this already, but I'd be willing to write up
an RFC describing a proposed approach, and to help implement it if you wish.
I dream of the day I no longer have to answer questions about why query
performance is often terrible through OGR. :-)

Best regards,
-- 
---------------------------------------+--------------------------------------
I set the clouds in motion - turn up   | Frank Warmerdam, warmerdam at pobox.com
light and sound - activate the windows | http://pobox.com/~warmerdam
and watch the world go round - Rush    | President OSGeo, http://osgeo.org