MS RFC 22: Data adapter support

[Frank Warmerdam]

Tamas Szekeres wrote:
> Since the data adapter can take control over all of the functions to
> the data provider
> i think it's much enough to populate the layer level cache with
> features. Since we will
> retain the feature of at least the previous rendering (or more). The 
> subsequent
> queryByXXX can be handled from that cache.

Tamas,

I have reviewed the query implementation, and functions like
msQueryByAttributes() currently just install filters and spatial
selections, do a whole bunch of vanilla msLayerNextShape() calls,
and record the id's.  So yes, caching all the features of the last
result set in a data adapter would accomplish roughly the same thing
as just storing the features (rather than just the feature ids) in the
resultCacheObj.

So, I agree that the mechanism from RFC 22 could provide for the
feature caching needed for query optimization.

I'm not really completely certain how it would work in practice though.
I'm not sure it would be good to keep around features fetched for drawing
purposes.  I had in the past assumed we would just store features instead
of feature ids in the resultCacheObj - in which case the caching would
only affect queries, and wouldn't increase memory use during the normal
render loop.

>> I'd add we could accomplish something similar now just by having "plugin
>> layers" or other specialized layers that refer to other layers are their
>> raw data source.  So, essentially chaining layers instead of maintaining
>> a stack of adapters.
> 
> Since the data adapter can take control over all of the functions to
> the data provider I'm
> considering the cascaded layer philosophy would be equivalent from the
> programmer's
> perspective. And I think that the actual problem could be solved by each.
> However the latter will bring in extra complexity by creating and
> additional internal
> layer inside a provider. And also I can't really imagine how this
> solution could be
> controlled by the mapfile or by the mapscript API repectively.

Note, I was suggesting that "adaptor layers" would reference another
raw feature data source layer by name, rather than nesting it within
the layerObj.  This is analygous to how the tileindexes are referenced
from other layers.

> Theoretically I would write a builtin support for caching features
> from any of the
> providers but i would not want to implement a new provider eg.
> MS_FEATURECACHE_PROVIDER along with MS_OGR or the others. This problem
> requires to extend the functionality of an existing provider not to
> write a new one.

I'm afraid I didn't understand this.

> As far as I know currently the tileindex support is implemented by the
> providers separately.
> I cannot see an uniformized decoupled implementation for this problem.

Use of the tileindex is currently embedded in some of the drivers
(raster, shape, ogr) but by use of "tile index layers" it is possible
to have a raster layer for which the tileindex comes from a postgis layer
(for instance).

>> And of course, currently, most "feature transformation" type operations
>> are accomplished using mapscript.
>>
> 
> I consider there operations are less effective than binding the work 
> into the
> core mapserver code. I guess it would require to create inline layers
> on the map.

I'm not sure why binding the work into the core mapserver code is
more effective.  When I spoke of Mapscript, I was meaning a case
where mapscript is used explicitly for drawing the features.  So
a mapscript loop can query the features from the layer, and then render
each with appropriate per-feature draw calls.  In this "mapscript owns
the rendering loop" approach any arbitrary transformation can be
applied.

Best regards,
-- 
---------------------------------------+--------------------------------------
I set the clouds in motion - turn up   | Frank Warmerdam, warmerdam at pobox.com
light and sound - activate the windows | http://pobox.com/~warmerdam
and watch the world go round - Rush    | President OSGeo, http://osgeo.org