[mapserver-dev] Call for discussion on MS RFC 114: Faster retrieval of shape count

Wed Feb 10 04:34:00 PST 2016

Hi,

I mean both server side and client side maxFeatures settings. As I read the standard the &resultType=hits should return a "numberOfFeatures" that is equal to the number of the features which are really returned if user makes the request as &resultType=results. But if implemented that way users have no means to check the total count of features belonging to the featureType if that count is bigger than the server side maxFeatures limit. So the "wfs_ignore_server_maxfeatures_for_hits" would be useful to have but I believe that it is not in the WFS standard. I may be wrong, this would not be the first time.

-Jukka-

Even Rouault wrote:

> Hi Jukka,

> 
> I am thinking about the metadata setting “wfs_ignore_maxfeatures_for_hits”.
> While it is certainly useful for the users sometimes I believe that 
> ignoring maxfeatures is still not WFS standard compliant.

Perhaps there is an ambiguity in the name since MAXFEATURES is also a WFS 1.1 parameter. "wfs_ignore_maxfeatures_for_hits" is aimed at ignoring the *server* "wfs_maxfeatures" setting, not the client one.

So perhaps "wfs_ignore_server_maxfeatures_for_hits" ?

Client MAXFEATURES will always be honoured (if no greater than server
"wfs_maxfeatures")

> Quotation from
> WFS 1.1.0 standard:
> 
> "In this way  [by using resultType=hits] a client may obtain a count 
> of the number of features that a query would return without having to 
> incur the cost of transmitting the entire result set."

MapServer current behaviour was non compliant in case a "wfs_maxfeatures" was defined in the mapfile. With this RFC, in the PostGIS case, it will now be compliant by default. We could default "wfs_ignore_maxfeatures_for_hits" to "true" for other providers as well, but there might be a performance hit : 
imagine querying a huge shapefile, OGR datasource (and also for now Oracle,
MSSQL) with > millions of records. This would mean all those records would be retrieved by MapServer to count them. Administrators might want to be able to avoid that.

> 
> If ignore maxfeatures is set in the metadata it makes that WFS service 
> permanently non-compliant.

non-compliant if it is set to "false".

This RFC makes MapServer compliant for PostGIS by default. And for other backends if "wfs_ignore_maxfeatures_for_hits" is explicitly set to "true". 
That seems a good compromise to me.

> I wonder if we could have a vendor parameter like 
> "&feature_count=total" for controlling which count &resultType=hits 
> will return.  For end user both counts can be valuable: total count 
> for knowing the size of the layer and the count that takes maxFeatures 
> into account the maximum size that can be used for paging.

- "total count for knowing the size of the layer":  that's what you get if "wfs_ignore_maxfeatures_for_hits" is set to "true" (explictly or implictly)
- "the count that takes maxFeatures into account the maximum size that can be used for paging" : given the above total count and the WFS 1.1 "DefaultMaxFeatures" / WFS 2.0 "CountDefault" values returned in GetCapabilities response, you can know if paging will be needed.

I don't think we need & want a vendor parameter (there's already enough complexity with the combinations of client MAXFEATURES parameter and server side limitations I think). The purpose of "wfs_ignore_server_maxfeatures_for_hits" and "wfs_maxfeatures" is to control how much we want a user request to be able to make the data backend busy, sometimes at the expense of strict standard conformance. Users should not be able to override that.

Does all the above make sense ?

Even

> 
> -Jukka Rahkonen-
> 
> Even Rouault wrote:
> > Hi Steve,
> > 
> >> Hi Even: I take it that "the_normal_sql_query" could also include 
> >> the translated filter described by a layers FILTER/FILTERITEM combo 
> >> or the NATIVE_FILTER processing option, correct?
> > 
> > Hum, I was convinced that my implementation in the PostGIS driver 
> > already did that, but it didn't ! And by further checking, I also 
> > saw I missed that
> 
> msQueryByFilter() could be called indirectly by the WFS code through 
> FLTxxxxx().
> 
> > Those 2 issues are now fixed with :
> https://github.com/rouault/mapserver/commit/4557afed33f92d22c969282d22
> dacf0
> 27e5e7387
> https://github.com/rouault/mapserver/commit/28354fe4bd5f5c3058a87ebc31
> d55c
> a7905388f3
> 
> > I've also added new test cases in msautotest:
> https://github.com/mapserver/msautotest/commit/05e8264ce737a795d81f1a8
> 7614f
> a5c87399de91
> 
> >> The only other gotcha is how (or if) to deal with class-level templates.
> >> The presence of a template is used to decide if a feature is to be
> >>
> >>processed as part of query operation and, in theory, if it should be
> >>
> >> counted. Typically one would configure a template at the 
> >> LAYER-level
> > 
> >> but it's also possible to define CLASS-level templates. For example:
> > CLASSITEM 'column'
> > 
> >> CLASS
> >> 
> >>   EXRESSION 'foo'
> >>   TEMPLATE 'void'
> >>   ...
> >> 
> >> END
> >> CLASS
> >> 
> >>   EXRESSION 'bar'
> >>   # no template
> >>   ...
> >> 
> >> END
> >> 
> >> In this case you wouldn't know if you should count a feature until 
> >> you know what CLASS it is. Granted this is an older configuration 
> >> option that probably isn't widely used but it does exist. Perhaps 
> >> it's currently broken anyway, I haven't tested. Just not sure how 
> >> to deal with it. Perhaps it just needs to be noted as a 
> >> configuration exception folks need to avoid when setting up WFS. 
> >> I'd be happy to see that option removed in the next version - it 
> >> adds unnecessary complexity to the query code. You can see the 
> >> affect in msIsLayerQueryable() in mapquery.c  among other places in that source file.
> > 
> > I indeed see such stuff in msQueryByRect() and didn't know what it 
> > was supposed to do. In fact, the WFS code has always forced a dummy 
> > "ttt.html" layer template (if no layer template is defined), so 
> > class templates are de-facto unused in WFS (since the msQueryXXXX() 
> > functions do no test class templates if a layer template is 
> > defined). And the
> > msLayerGetShapeCount() optimization path is only taken if a layer 
> > template is defined. So for non-WFS case, map-
> >
> >query.only_cache_result_count == 1 and class templates will still 
> >work as
> 
> before.
> 
> > I've modified the RFC with the above changes and precisions :
> https://github.com/mapserver/docs/commit/76dfe8cb73c9c3de2f2e878e7c856
> dedc7
> a10c28
> 
> > Thanks a lot for your very much appreciated input !
> > 
> > Unrelated note: my latest commit fails in renderers/wmsclient.map 
> > because
> 
> http://demo.mapserver.org/cgi-
> bin/mapserv?map=/osgeo/mapserver/msautotest/world/world.map&service=WM
> S&req
> uest=GetCapabilities doesn't respond.
> 
> > Even
> > 
> >> Steve
> > 
> > -----Original Message-----
> > From: mapserver-dev [mailto:mapserver-dev-bounces at lists.osgeo.org] 
> > On Behalf Of Even Rouault Sent: Monday, February 08, 2016 4:58 AM
> > To: MapServer Dev Mailing List <mapserver-dev at lists.osgeo.org>
> > Subject: [mapserver-dev] Call for discussion on MS RFC 114: Faster 
> > retrieval of shape count
> > 
> > Hi,
> > 
> > Your comments on the following RFC are welcome :
> > http://mapserver.org/development/rfc/ms-rfc-114.html
> > 
> > Even
> 
> --
> Spatialys - Geospatial professional services http://www.spatialys.com 
> _______________________________________________ mapserver-dev mailing 
> list mapserver-dev at lists.osgeo.org 
> http://lists.osgeo.org/mailman/listinfo/mapserver-dev

--
Spatialys - Geospatial professional services http://www.spatialys.com