MS RFC 22: Data adapter support

[Stephen Woodbridge]

Tamas,

I think some of the generic issues involved with a cache are caste into 
the context of this RFC follow. I think answers to questions like this 
will help the rfc provide a high level overview of how this will work 
and deal with specific cache related issues.

1. How large can it get? Can I configure a max size? How?
2. What happens to the behavior when it fills up?
3. How are items expired from the cache?
4. What happens when the cache gets fragmented?
5. What happens when there is not room in the cache to add to it? Either 
it is fragmented or full?
6. There has been some implementation of layer/feature? security, I 
thought, how will the cache deal with that?
7. Can/do items have a Expiration time set on them? after which they are 
invalid and can be purged or assumed to be purged?
8. Is a cache - mapfile specific or mapserver instance specific or user 
session specific? How will this be handled from a config and 
installation point of view? In a live production site with multiple 
users, multiple mapfiles, and multiple mapserver instances how does all 
this work? How to the requests get to their respective cache instance.

Answers to many of these questions probably represent config options for 
the cache, where would these be configured on the installation server?

I did not check the rfc, you may have addressed some of these already, 
this is a dump of the problems I have had to consider working with other 
caching applications.

-Steve W

Tamas Szekeres wrote:
> Frank,
> 
>>
>> I'm not really completely certain how it would work in practice though.
>> I'm not sure it would be good to keep around features fetched for drawing
>> purposes.  I had in the past assumed we would just store features instead
>> of feature ids in the resultCacheObj - in which case the caching would
>> only affect queries, and wouldn't increase memory use during the normal
>> render loop.
>>
> 
> I would support caching not only for the feature query but also for
> the subsequent
> renderings as well. Therefore at least the features of the previous
> rendering would be
> retained in the cache. Alternatively the user could configure the
> adapter for fetching
> features for an enlarged rendering area or for the entire area covered
> by the layer.
> The RFC for the feature cache will address the applications being
> capable to maintain
> long lived internal reference to the map object. This kind of
> application will tolerate the
> increased memory footprint in exchange for the higher rendering and query
> performance.
> I would also support adapter dependent custom actions to be performed 
> using the
> mapscript API such as loading the features into the cache. This action 
> could be
> implemented thread safely so prefetching the features could be done even 
> from a
> background thread.
> 
>>
>> Note, I was suggesting that "adaptor layers" would reference another
>> raw feature data source layer by name, rather than nesting it within
>> the layerObj.  This is analygous to how the tileindexes are referenced
>> from other layers.
>>
> 
> At this point I cannot figure out how this approach would be realized.
>  From the user's
> perspective how it will affect the mapfile configuration and the
> mapscript API. Will it
> change the API in a logical or rather a constrainted manner?
> And how much amount of code should be changed to implement this feature?
> 
>> > Theoretically I would write a builtin support for caching features
>> > from any of the
>> > providers but i would not want to implement a new provider eg.
>> > MS_FEATURECACHE_PROVIDER along with MS_OGR or the others. This problem
>> > requires to extend the functionality of an existing provider not to
>> > write a new one.
>>
>> I'm afraid I didn't understand this.
> 
> Well I have just mentioned that I could not imagne the suggested
> solution easily.
> But now, I would consider a solution providing to specify one or more
> layer being contained
> by another layer. This nested approach could be implemented for the
> mapfile configuration
> and the mapscript API as well like:
> 
> LAYER
>  CONNECTIONTYPE CUSTOM_LAYER
>  ...
>  LAYER
>    CONNETIONTYPE OGR
>    ...
>  END
>  LAYER
>    CONNECTIONTYPE POSTGIS
>  END
> END
> 
> I'm considering this solution is different from my proposal, and would
> require to change a similar
> amount of the existing code. This approach would give a solution for
> different problems
> (like implementing provider independent joins for example). Recall
> that originally I would like
> to add functionality to and existing provider. I can see a slight
> analogue to the difference between
> the inheritance and the containment from the OO terminology.
> 
>>
>> I'm not sure why binding the work into the core mapserver code is
>> more effective.  When I spoke of Mapscript, I was meaning a case
>> where mapscript is used explicitly for drawing the features.  So
>> a mapscript loop can query the features from the layer, and then render
>> each with appropriate per-feature draw calls.  In this "mapscript owns
>> the rendering loop" approach any arbitrary transformation can be
>> applied.
>>
> 
> Do you mean creating an inline layer and feeding the features into it
> on the fly?
> How can the implementor provide the reusability and the configurability of
> the implementation if needed?
> 
> 
> Best Regards,
> 
> Tamas