MS RFC 22a: Feature cache for long running processes and query processing (update)

[Tamas Szekeres]

2007/6/26, Frank Warmerdam <warmerdam at pobox.com>:
>
> I'm concerned about thread safety issues if a single cache is being
> used by multiple threads at the same time.  Are you planning on fine
> grained locking around each request for a shape from the cache?
> Basically, I think fully thread-shared cache access is going to be
> dangerous and should not be implemented.
>

Certainly I would protect against the multiple access of the common
data by using locks. However I still haven't consider all of the
possibilities about the desired locking strategy. Apparenly the whole
transaction against the common data should be atomic not only the
feature retrievals. I should also consider to use "shared" locks among
the read only operations and sometimes create local copies of a subset
of shapes for each layer. I would rather delay these considerations
after the 5.0 release.
Recall that we have a feature/code freeze deadline at July 9th.

> That aside, the connection pooling approach is still not suitable
> because the CONNECTION string for stuff like CACHE layers is not
> a very unique name.  We could just modify the pooling API so that
> the "connection" name is passed in.  Then for CACHE layers this
> could be some more unique name.  Alternatively we could just declare
> that folks wanting long lived caches (presumably using the usual
> PROCESSING "CLOSE=DEFERRED" line) should take great care to use
> unique layer names in their CONNECTION string.
>

Using a parameter instead of the implicit use of the layer->connection
would not be enough if we would want to use the same object
simultaneously. In addition, theoretically, I still don't feel the
exact match between the connection pooling and the preservation of an
internal state. The first one might be a subset of the second (?)

>
> 1) I wish there were specific sections in the RFC giving detailed
>     docs for each of the new data providers (CACHE, LAYERFILTER and
>     GEOMTRANS I believe).  I am left with a vague sense that I'm not
>     seeing all the layerfilter and geomtrans operators by scanning
>     the examples.
>

Formerly I've been thinking to implement layerfilter and geomtrans at
a later time. However it's quite an useful to introduce them now, so
as to show up the flexibility of the approach at the beginnings.
However this addition is newish enough not to grave everything into
the stone at the moment.

> 2) I am nervous about the "introduced language" for describing
>     geometry filtering operations.  I'd be inclined to make this
>     particular provider unofficial for 5.0 with the understanding
>     that we might want to overhaul the syntax for 5.2.   In particular
>     I wish we had a more functional (perhaps Simple Features for SQL-like)
>     syntax for geometry operations.
>

I like the idea behind that. At least I should establish an approach
to evaluate the geometry expressions. That would eliminate the need of
using separate operation specific parameters like buffer_width.
Originally I've been planning to support only 1 operation per layer
and linking the layers to each other for nesting that operations. But
it's not a strong requirement.

> 3) I have a nervous sense that nested layers are going to cause us
>     unexpected problems.
>

I don't see any issues about nesting the layers itself. The mapserver
core normally doesn't do other with the nested layers as creating and
freeing when specified. The only issue is that how these new providers
use those layers but it's the problem of the implementor of that
providers ;-) At the moment I don't know about significant problems
that cannot be treated. However that might be a few difference how the
various providers communicate with the outer world but it's a subject
of  a standardization rather than keep it divergent.

> 4) How are nested layers manipulated in mapscript?  I'd like to see
>     this addressed in the RFC.
>

I would keep on using the insertLayer(layerObj layer, int index) and
layerObj removeLayer(int index) approach on the layerObj. And create a
getSubLayer(string path) method on the mapObj which would return a
layerObj.

Best regards,

Tamas