[UMN_MAPSERVER-USERS] MS RFC 22: Data adapter support

[Steve Lime]

Can folks please cc the dev list (and vice versa)  on this thread. We'd
got different posts in two places so it's hard to
follow...

Steve

>>> Stephen Woodbridge <woodbri at SWOODBRIDGE.COM> 10/24/2006 8:43:23 AM
>>>
Tamas,

I think some of the generic issues involved with a cache are caste into

the context of this RFC follow. I think answers to questions like this

will help the rfc provide a high level overview of how this will work 
and deal with specific cache related issues.

1. How large can it get? Can I configure a max size? How?
2. What happens to the behavior when it fills up?
3. How are items expired from the cache?
4. What happens when the cache gets fragmented?
5. What happens when there is not room in the cache to add to it?
Either 
it is fragmented or full?
6. There has been some implementation of layer/feature? security, I 
thought, how will the cache deal with that?
7. Can/do items have a Expiration time set on them? after which they
are 
invalid and can be purged or assumed to be purged?
8. Is a cache - mapfile specific or mapserver instance specific or user

session specific? How will this be handled from a config and 
installation point of view? In a live production site with multiple 
users, multiple mapfiles, and multiple mapserver instances how does all

this work? How to the requests get to their respective cache instance.

Answers to many of these questions probably represent config options
for 
the cache, where would these be configured on the installation server?

I did not check the rfc, you may have addressed some of these already,

this is a dump of the problems I have had to consider working with
other 
caching applications.

-Steve W

Tamas Szekeres wrote:
> Frank,
> 
>>
>> I'm not really completely certain how it would work in practice
though.
>> I'm not sure it would be good to keep around features fetched for
drawing
>> purposes.  I had in the past assumed we would just store features
instead
>> of feature ids in the resultCacheObj - in which case the caching
would
>> only affect queries, and wouldn't increase memory use during the
normal
>> render loop.
>>
> 
> I would support caching not only for the feature query but also for
> the subsequent
> renderings as well. Therefore at least the features of the previous
> rendering would be
> retained in the cache. Alternatively the user could configure the
> adapter for fetching
> features for an enlarged rendering area or for the entire area
covered
> by the layer.
> The RFC for the feature cache will address the applications being
> capable to maintain
> long lived internal reference to the map object. This kind of
> application will tolerate the
> increased memory footprint in exchange for the higher rendering and
query
> performance.
> I would also support adapter dependent custom actions to be performed

> using the
> mapscript API such as loading the features into the cache. This
action 
> could be
> implemented thread safely so prefetching the features could be done
even 
> from a
> background thread.
> 
>>
>> Note, I was suggesting that "adaptor layers" would reference
another
>> raw feature data source layer by name, rather than nesting it
within
>> the layerObj.  This is analygous to how the tileindexes are
referenced
>> from other layers.
>>
> 
> At this point I cannot figure out how this approach would be
realized.
>  From the user's
> perspective how it will affect the mapfile configuration and the
> mapscript API. Will it
> change the API in a logical or rather a constrainted manner?
> And how much amount of code should be changed to implement this
feature?
> 
>> > Theoretically I would write a builtin support for caching
features
>> > from any of the
>> > providers but i would not want to implement a new provider eg.
>> > MS_FEATURECACHE_PROVIDER along with MS_OGR or the others. This
problem
>> > requires to extend the functionality of an existing provider not
to
>> > write a new one.
>>
>> I'm afraid I didn't understand this.
> 
> Well I have just mentioned that I could not imagne the suggested
> solution easily.
> But now, I would consider a solution providing to specify one or
more
> layer being contained
> by another layer. This nested approach could be implemented for the
> mapfile configuration
> and the mapscript API as well like:
> 
> LAYER
>  CONNECTIONTYPE CUSTOM_LAYER
>  ...
>  LAYER
>    CONNETIONTYPE OGR
>    ...
>  END
>  LAYER
>    CONNECTIONTYPE POSTGIS
>  END
> END
> 
> I'm considering this solution is different from my proposal, and
would
> require to change a similar
> amount of the existing code. This approach would give a solution for
> different problems
> (like implementing provider independent joins for example). Recall
> that originally I would like
> to add functionality to and existing provider. I can see a slight
> analogue to the difference between
> the inheritance and the containment from the OO terminology.
> 
>>
>> I'm not sure why binding the work into the core mapserver code is
>> more effective.  When I spoke of Mapscript, I was meaning a case
>> where mapscript is used explicitly for drawing the features.  So
>> a mapscript loop can query the features from the layer, and then
render
>> each with appropriate per-feature draw calls.  In this "mapscript
owns
>> the rendering loop" approach any arbitrary transformation can be
>> applied.
>>
> 
> Do you mean creating an inline layer and feeding the features into
it
> on the fly?
> How can the implementor provide the reusability and the
configurability of
> the implementation if needed?
> 
> 
> Best Regards,
> 
> Tamas