MS RFC 22a: Feature cache for long running processes and query processing (update)

[Tamas Szekeres]

2007/6/27, Daniel Morissette <dmorissette at mapgears.com>:
> Tamas, and all,
>
> I finally managed to walk through RFC-22a. I can't say that I understand
> it all but I think I get the idea. The new options sound cool, but my
> first reaction is that it's not easy to understand how this works and
> what the implications are... and if it's hard for a developer to
> understand then imagine what it will be for the average users. I'm
> starting to miss the simplicity that's always been a focus in MapServer.
>

Daniel,


I personally don't favour the "average user" terminology in this area.
I consider the
users as talented developers who spare no trouble to read some documentation
than might help to solve that particular problem they have. If they
have no motivation
to do so, then they will possibly have to accept the problem or wrap a
fair amount
of code around mapserver/mapscript to have something working.
In this RFC not only the 2 pass query problem was addressed. I've also
taken up some
additional issues that might require the user to build a "shell of
code" around mapserver but
it should be internally supported. (like applying geometric
transformations on the features,
representing the query results as separate layers, providing a subset
of the features
with spatial and attribute filtering and allow them all to be cascaded
to solve a number
of complex rendering issues inline) The reason I could do them all is
that I rely on the layer
vtable interface which is the only generalized interface around the
various layer providers I
could utilize.
I think that this addition is quite simple comparing with the
complexity of the problems
it may solve. There's no new mapserver entities have been introduced. We can
continue to think about layers as the providers of a set of features
that will eventually
be rendered on the map. The most significant new issue is to consider how the
set of the features will change among the layers have been connected,
but I think
this approach would provide a higher level of abstraction to solve the
problems.
It fairly accepted in many area to properly connect simple reusable
entities for solving
complex problems instead of creating a mixed application specific
implementation.

> More importantly, I started asking myself how this RFC solved the
> initial problem which is the performance hit related to double-pass
> queries with several providers (did we forget that?).
>

All of the layers suffering from the double-pass issue would be
wrapped by a CACHE
layer. That layer would store all of the features of a previous
rendering or query.
The proposed solution would either store all of the features of the
extent specified
by the previous WhichShapes call or the user could choose the option
to retrieve all of the shapes within the actual map extent. If a subsequent
WhichShapes has an extent within the previous one, then no access
to the original provider will be happen.
In the 2-nd phase the GetShape calls based on the resultcache will be
served by the CACHE layer using hashtable lookups.

> If I understood the RFC correctly, the proposed solution to this problem
> will be to embed all layers inside a CACHE wrapper layer, correct? So by
> default all postgis/oracle/sde/etc connections will still suffer from
> the double-pass query problem unless we add the CACHE layer around them,
> right? This adds an additional thing that users needs to be aware of in
> order to tune their app.

We have only one common interface around the providers, represented by the
vtable operations. If we want to hook into this option we shall have to create
additional providers and link them to each other, definitely. The
benefit of this
option is that we can leave the existing providers, rendering and query
codes intact. And we can present an arbitrary number of further
shape processing options in the future based on this architecture.

Or alternatively we could focus only on the 2 pass query problem by not
utilizing the vtable. It will possibly require either to modify all of
the mapserver
code involved in the query operations or modify all of the providers suffering
from this particular problem. This might require a large amount of changes
in the existing code and would solve at most 20% of the problems I've
addressed.