Thoughts on a feature cache for queries...

[Steve Lime]

Any ideas and help appreciated... I agree that making this tunable makes sense. That pushes
us away from the "queries as a layer" concept I think. I do think that in most cases a linear
traversal is the most likely access method.

Steve

>>> On 6/6/2007 at 10:13 AM, in message <4666CF0A.4070600 at pobox.com>, Frank
Warmerdam <warmerdam at pobox.com> wrote:
> Steve Lime wrote:
>> Hi all: What would folks think about implementing a simple feature cache for
>> storing query results? I know Tamas has something grander planned but it
>> seems a ways off and implementation with queries may be problematic anyway.
>> We could use the featureList structures that form a linked list containing
>> shapeObj's. I've always been a bit hesitant to do this because with large
>> datasets you could use a ton of memory, but the performance bottlenecks with
>> the two-pass query probably out weigh that problem. If folks are somewhat
>> positive about the idea I'd be glad to work up an RFC...
> 
> Steve,
> 
> I think this is an important development and worth an RFC to work out the
> details.
> 
> I'm hesitant about using the featureList linked list approach, as that is
> hard to access quickly except by linear traversal.
> 
> I am imagining some sort of feature cache in the layerObj which
> msLayerGetShape() would first check before calling the provider specific
> LayerGetShape() function.  The cache could use a hash table approach or
> perhaps an ordered array allowing binary searches.
> 
> Then we could choose whether or not to populate this cache with query
> results based on the provider type or perhaps some flag in the layer object
> from the map file.  That way the mechanism wouldn't be used for layer types,
> such as shapefiles, that allow fast by-index lookups anyways.
> 
> I am kicking myself for doing this already, but I'd be willing to write up
> an RFC describing a proposed approach, and to help implement it if you wish.
> I dream of the day I no longer have to answer questions about why query
> performance is often terrible through OGR. :-)
> 
> Best regards,