[mapserver-users] ONE PASS QUERY (RFC 52) - FEATURE OR BUG?

[Lime, Steve D (DNR)]

I think that would probably eliminate the WFS issues we saw in 5.6 where OGC filters didn't map cleanly to a core query as we'd essentially be adding a new and more flexible core query. I'm not so sure that
it would address a common use case for old behavior. For example, it seems that folks like to start with a core query against a number of layers (e.g. find me all restaurants, bars, and coffee shops within a
certain bbox) and then they (through subsequent UI interactions) augment that original set of results by adding or removing features in a number of layers based on new queries, often a point query. I 
think that would be very hard to manage through a single filter. It's easy to do, however, via a list of feature IDs.

I still think adding a filterObj and a msLayerWhichShapesFiltered() is a good idea and want to see this in 6.0.

In addition, I was thinking that:

 1) we could use shapeindex to hold a global feature ID (OID, row id, etc...), and tileindex (for non-tiled shapefile/raster layers) to hold a result set specific row id
 2) driver specific version of msLayerNextShape(...) would set shapeindex and optionally tileindex (for example, the shapefile driver would only set the former, postgis would set both)
 3) driver specific versions of msLayerGetShape(...) would be charged with making the decision on doing either a random access query (select ... where oid=x) or leveraging the existing result set based 
     on the passed tileIndex (msLayerResultsGetShape(...) goes away)
 4) resurrect the old query file writer (in addition to the new one). That code wrote the shapeindex and tileindex but we'd only cache the tileindex for tiled shapefile/raster layers
 5) resurrect the old query file reader and it would load a query in a state that wouldn't have the tileindexes so old, slow processing would result for RDBMS layers

Knowing when the result set processing (e.g. via WFS, templates and query maps) should set things up for old vs. new is a bit tricky. Right now there's a flag set in the result cache that I believe only
the GML writer respects. It shows that it's possible to support both worlds though. That flag could trigger old/new file IO and result cache processing. 

I need to resolve this with the parallel discussion Tamas and Frank were having.

My 2 cents anyway...

Steve

________________________________________
From: mapserver-users-bounces at lists.osgeo.org [mapserver-users-bounces at lists.osgeo.org] On Behalf Of Paul Ramsey [pramsey at cleverelephant.ca]
Sent: Monday, March 22, 2010 11:25 PM
To: Lime, Steve D (DNR)
Cc: mapserver-users at lists.osgeo.org; BrainDrain
Subject: Re: [mapserver-users] ONE PASS QUERY (RFC 52) - FEATURE OR BUG?

Is it better to back out to the old behaviour or would defining a
filter object that allows complex query logic meet the need in a more
direct way? (Ie, is running multiple queries a feature or a workaround
for an even older limitation?)

P.

On Mon, Mar 22, 2010 at 9:21 PM, Lime, Steve D (DNR)
<Steve.Lime at state.mn.us> wrote:
> I think we're in need of a RFC 52a. Clearly the compound query handling the old approach afforded is of value to a group of users and that wasn't accounted for in the initial RFC. The work around Assefa had to do with WFS and a certain subset of OGC filters at the sprint is evidence that the approach was even used in the core code (I wasn't aware of that at the time). We (in 5.6.3) developed a work around that retains two sets of indexes one suitable for random access and one for a specific result set (e.g. cursor). It uses the already present tileindex property of a shapeObj to store the latter. I think we can have the best of both here by storing the two indexes and potentially we can revert to a single getShape() function in MapScript and revive the old queryfile format as a option as well. Just needs to be planned for now that the full impacts are better understood.
>
> Steve
> ________________________________________
> From: mapserver-users-bounces at lists.osgeo.org [mapserver-users-bounces at lists.osgeo.org] On Behalf Of Frank Warmerdam [warmerdam at pobox.com]
> Sent: Monday, March 22, 2010 10:16 PM
> To: Tamas Szekeres
> Cc: mapserver-users at lists.osgeo.org; BrainDrain
> Subject: Re: [mapserver-users] ONE PASS QUERY (RFC 52) - FEATURE OR BUG?
>
> Tamas Szekeres wrote:
>> In my understanding with the original approach the driver should:
>>
>> 1. Retain the result set of the queries at the layer (ie. in the
>> layerinfo structure) until the layer is open and no subsequent
>> whichShapes is called to 'invalidate' the query.
>
> Tamas,
>
> Your point here is that the query result should live until
> invalidated by another whichShapes, right?  I would agree with
> that, but draw on a layer does do a whichShapes, right?  So a
> draw is expected to invalidate a query, right?
>
>> 2. Provide such index in shapeObj which would allow to retieve in a
>> subsequent resultsGetShape within the result set.
>
> ok
>
>> 3. Retain the random access behaviour (getShape) for backward
>> compatibility in parallel to resultsGetShape.
>
> ok
>
>> Since the RFC doesn't contain explicit note about the opposite, the
>> drivers should also:
>>
>> 4. Preserve the behaviour of keeping separate set of results for
>> separate layer instances. In this regard a query on one layer should not
>> invalidate the results for a different layer instance of the same driver.
>
> This seems to be a lot to expect.  We go to significant effort with
> the connection pooling to allow reuse of a connection for different
> layers, and in effect in many drivers this connection also carries
> a bunch of context with it.  Certainly in the case of OGR an
> OGRLayer retains a concept of current query result, but it can
> be invalidated by lots of operations other than ResetReading() and
> GetNextFeature().  I would imagine this is true to a greater or
> lesser to other drivers that pool connections.
>
>> 5. Creating a clone of a layer should provide to use a separate query
>> (by keeping the results intact on the original layer). This would be
>> essential for msDrawQueryLayer to work when drawing the background
>> before the highlighted features.
>
> This is also quite impractical for some implementations - certainly
> for OGR.
>
>> 6. Using a drawQuery should not invalidate the results of a previous query.
>
> I don't know much about drawQuery but it does seem plausible to ask
> that drawQuery should not invalidate the query it is drawing.
>
>> 7. Drawing the map should not invalidate the results of a previous query.
>
> But drawing maps uses the feature access machinery like whichShapes
> doesn't it?  How can we expect map drawing not to invalidate a query?
>
> In retrospect, I'm not all that confident that we really considered
> the impact of RFC 52 on use cases such as those you raise.  I
> certainly didn't understand these impacts.  What is less clear to
> me is where to go from here.  RFC 52 was put in place because the
> old approach was giving terrible performance in some cases.
> But if we put the expectations you list into place there is no
> way it can be made fast on OGR short of maintaining distinct
> OGRDataset instances for each query in addition to the one used to
> draw the layer.  This could cause various performance and
> resource problems.
>
> Best regards,
> --
> ---------------------------------------+--------------------------------------
> I set the clouds in motion - turn up   | Frank Warmerdam, warmerdam at pobox.com
> light and sound - activate the windows | http://pobox.com/~warmerdam
> and watch the world go round - Rush    | Geospatial Programmer for Rent
>
> _______________________________________________
> mapserver-users mailing list
> mapserver-users at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/mapserver-users
> _______________________________________________
> mapserver-users mailing list
> mapserver-users at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/mapserver-users
>
_______________________________________________
mapserver-users mailing list
mapserver-users at lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/mapserver-users