MS RFC 22a: Feature cache for long running processes and query processing (update)

[Tamas Szekeres]

2007/6/25, Howard Butler <hobu.inc at gmail.com>:

> - The proposal really continues our path to PROCESSING option
> hell :)  It would be nice if we could somehow make the parser not
> cough over (new) things it doesn't recognize (too late to go back in
> time to do that though).  I have no problem with what is proposed,
> this is just more a comment on our current free-form, alternative
> universe of mapfile syntax we have going on.

Howard,

I'd like to keep this proposal as simple as possible, and I didn't
want to bring in much of fairly unrelated problems to solve in one
package. It seems to me that the PROCESSING option is widely used
under the hood, though it haven't been declared as the only suggested
solution for specifying provider related data.
Yes, indeed, we might want to deal with the desired solution how to
specify the provider specific data, but it should be a subject of a
separate RFC. In this particular case I have no problem with the
PROCESSING option itself, because the values are read rarely and does
not affect the overall performance of the providers.

> - The proposal doesn't state whether or not a cache layer will
> participate in the connection pool.  Is the handle to a cache layer
> intended to be kept alive like other connection handles across
> requests (ala FastCGI or some sort of always-on MapScript stuff)?
> Would it be reasonable to keep a connection handle to the
> "datasource" of all of the cache layers in the connection pool and
> then pull out references to already active cache layers?

Most of the proposed providers keep long term references to the source
layers, however there's not too much extra cost using the
msGetSubLayer lookups.
In my case it makes not much sense to have a global repository of the
layer references and make similar lookups based on that "connection
store".

The referenced layers can continue to be pooled using the existing
approach. For example if the caching provider connects to an SDE layer
we can continue to specify the CLOSE_CONNECTION=DEFER processing
option on the SDE layer.

Moreover it is supported that the proposed providers connect to the
same source layer. If that source layer is eventually a caching
provider, then the same cache will serve the features for each of the
connected layers.

> - Is there any chance we would ever serialize a cache layer to disk?

This option might raise some further issues to consider. At first we
should have an efficient option to serialize the features and decide
how to control the process of the serialization. Or should we begin
supporting write capabilities on the existing the providers and have
the source layers to save the features in some widely accepted
formats?
These additions should also be handled in separate RFC-s.

> - Will we ever do raster layer caching?  If so, MS_CACHE may not be
> what we want for a name for the enum...

Currently the raster renderings bypass the vtable mechanism and do the
drawings directly from the raster files. So this option would require
a different approach by having the source layer to draw on a
background image and later this image would be copied back to the
caller. At the moment I haven't consider the possible drawbacks of
such a solution. And I'm not really sure about the potential
performance increment related to this change. In any case this
addition would require a new provider.

> - I would be interested in exploring the consequences of sub layers
> for MapServer layers.  Are these only to be cache layers, or could
> they potentially be *any* layer type?

The sublayer approach is not bound tightly to the caching approach. We
can specify arbitrary number of sublayers without setting up
references to them. Those layers can be any kind of the existing
layers including the proposed layers and the plugin layers as well.

>    -- Also, are the sub layers references or copies (ala RFC 24)?
>

The sublayers are separate layer entities. And the hierarchy of the
layers can be specified in the mapfile or at the mapscript interface.

When the proposed providers establish a reference to another layer (to
a sublayer or a root layer) increment the reference counter of that
layer (according to RFC 24) so that the source  layer cannot be
destroyed behind the scenes.


Best regards,

Tamas