MS RFC 22: Data adapter support (void of interest?)

[Steve Lime]

>>>> "Tamas Szekeres" <szekerest at gmail.com> 10/26/2006 4:17:26 PM >>>
> Steve,

>> I was sort of sitting back and watching where the discussion with
Frank was going
>> to go since you both are amongst the most technically skilled and
then comment a bit.
>> I confess to being a bit lost in the technical details of RFC 22.
>>

> That's very kind of you. I consider the conversation between me and
Frank is
> closed. We have made a further illuminating talk on the IRC however
we have
> made our standpoints clear to each other.

Is Frank still a +0 or did you sway him... ;-)

> At this point me - as the proposer - would like to see how the
foundation of
> this RFC was succeeded. With the absence of notes I cannot see how
can
> I make the things more clear and see whether the subject of the
document is
> difficult to follow or not. (I think this was not an issue for
Frank)

I think it reads fine. Practical uses of the feature will help.

> One thing I did notice is that for queries to work "as is" the
adapter, specifically the
> cache adapter would need the ability to retrieve features by unique
ID or better
> yet member shapes would need to be cached by hand from the
msQueryBy... functions
> so we could pull them sequentially for presentation.
>

> Hmmm... I guess I couldn't understand this conclusion exactly. The
> most compelling behaviour of the cache adapter is to retrieve the
> features using the mapservers 'bulk' mode by utilizing WhichShapes
and NextShape 
> to the provider. All of the subsequent renderings and the QueryBy..
operations 
> could be handled from the cache. Recall that currently serving
QueryByShape requires 
> at least n+1 provider accesses and 2 layeropens where n is the number
of the features
 > found. And I am guessing even more improvements for the further
renderings by 
> using msDrawQueryMap for example.

My thought is that caching for queries is different than caching for
long running processes.
A query cache would in many cases stay resident for one execution only.
I don't think we
want to query the cache other than spatially. We'd always go back to
the provider for certain
queries (e.g. attribute). This is a discussion for the RFC for caching
though.

> I'm a bit disappointed about the inevitable copyShape when serving
from the cache
> since the caller has ownership of the features retrieved but I can be
reconciled to
> it for now.

For certain queries you don't have to copy the shape since no clipping
or coordinate 
conversion is necessary.

>> My thoughts on caching are that it is very much necessary. I was
thinking mostly
>> from a query perspective but do set the value in a long running
process. When I
>> had thought about implementation it I was thinking that caching
would be independent
>> of the underlying datasource. That is, a filter that would sit
between MapServer
> drawing and query functions.

> Wow, that's exactly what I have addressed.

>> That way we wouldn't have to change in the existing
>> code base at all, nor would virtual providers have to worry about
>> anything. Layer configuration for caching would be simple, STATUS
On/Off/Query,
>> SIZE/TUNING parameters and that's about it.

> Well, one of the most important conclusion of the conversation with
Frank was
> that even the most good-looking solution requires modification when
we are
> thinking of how to implement them.

> For me hooking into the vtable was the most trivial, but I also had
to make some
> changes as you can see.

> Furthermore I'm not keen to propose a solution restricted solely to
the
> caching problem. Instead would want to give a framework where other
> sorts of extensions to the the data provider could easily be done.

That's were the GEOS filtering got me thinking...

> Support for cascading the adapters is the other point where one can
> feel the solution a bit complex.

I dunno, being able to apply multiple adapters might make sense. For
example, if
one adapter is a cache then why wouldn't you want to potentially buffer
those
features?

> According to the ID-s I consider the providers not to keep the
feature
> indexes between the subsequent WhichShapes so I would restrict the
> contets of the cache to one retrieval at a given time. However I
would 
> allow to control how the cached area is determined (not always the
same 
> as the rendering area)

>> Your proposed solution goes well beyond the simple notions I had
been
>> thinking about. Also I was mulling over how to put GEOS functions in
the
>> pipeline (e.g. buffer a feature before drawing or querying with),
and could only come
>> up with using the PROCESSING tag so this would be better.

> I would leave the original provider alone with the PROCESSING
parameter so
> with regard to this the provider would behave similarly. And I would
also
> give a support for getting the shape by reference and changing them
> programmatically by the mapscript users for the subsequent
renderings.

I would advocate NOT using the PROCESSING parameter in favor of, say, a
buffering
adapter, or a general purpose GEOS adapter.

>> I think it would be helpful to write RFC 23 on the caching adapter
so
>> they can be considered in tandem, that is, a concrete example of the
benefits of
>> RFC 22.

> Agreed. But I wouldprevent the developers from a flood of the RFC-s
> as long as this one seems to be definitely unsupported.
> But it would be interesting to see some performance measures so I
will
> go ahead with it

Cool, appreaciate the willingness to dig in on a topic like this. Are
you going to do
so R&D first (e.g. get things working) or ??? There's probably more
interest in talking
about the caching than infrastructure- that's always the case.

>> Consider this more than mild support.

> Promising. Thanks.

> Best Regards,

> Tamas

Steve