[mapserver-dev] Call for discussion on MS RFC 114: Faster retrieval of shape count

Wed Feb 10 04:18:28 PST 2016

Hi Jukka,

> 
> I am thinking about the metadata setting “wfs_ignore_maxfeatures_for_hits”.
> While it is certainly useful for the users sometimes I believe that
> ignoring maxfeatures is still not WFS standard compliant.

Perhaps there is an ambiguity in the name since MAXFEATURES is also a WFS 1.1 
parameter. "wfs_ignore_maxfeatures_for_hits" is aimed at ignoring the *server* 
"wfs_maxfeatures" setting, not the client one.

So perhaps "wfs_ignore_server_maxfeatures_for_hits" ?

Client MAXFEATURES will always be honoured (if no greater than server 
"wfs_maxfeatures")

> Quotation from
> WFS 1.1.0 standard:
> 
> "In this way  [by using resultType=hits] a client may obtain a count of the
> number of features that a query would return without having to incur the
> cost of transmitting the entire result set."

MapServer current behaviour was non compliant in case a "wfs_maxfeatures" was 
defined in the mapfile. With this RFC, in the PostGIS case, it will now be 
compliant by default. We could default "wfs_ignore_maxfeatures_for_hits" to 
"true" for other providers as well, but there might be a performance hit : 
imagine querying a huge shapefile, OGR datasource (and also for now Oracle, 
MSSQL) with > millions of records. This would mean all those records would be 
retrieved by MapServer to count them. Administrators might want to be able to 
avoid that.

> 
> If ignore maxfeatures is set in the metadata it makes that WFS service
> permanently non-compliant.

non-compliant if it is set to "false".

This RFC makes MapServer compliant for PostGIS by default. And for other 
backends if "wfs_ignore_maxfeatures_for_hits" is explicitly set to "true". 
That seems a good compromise to me.

> I wonder if we could have a vendor parameter
> like "&feature_count=total" for controlling which count &resultType=hits
> will return.  For end user both counts can be valuable: total count for
> knowing the size of the layer and the count that takes maxFeatures into
> account the maximum size that can be used for paging.

- "total count for knowing the size of the layer":  that's what you get if 
"wfs_ignore_maxfeatures_for_hits" is set to "true" (explictly or implictly)
- "the count that takes maxFeatures into account the maximum size that can be 
used for paging" : given the above total count and the WFS 1.1 
"DefaultMaxFeatures" / WFS 2.0 "CountDefault" values returned in 
GetCapabilities response, you can know if paging will be needed.

I don't think we need & want a vendor parameter (there's already enough 
complexity with the combinations of client MAXFEATURES parameter and server 
side limitations I think). The purpose of 
"wfs_ignore_server_maxfeatures_for_hits" and "wfs_maxfeatures" is to control 
how much we want a user request to be able to make the data backend busy, 
sometimes at the expense of strict standard conformance. Users should not be 
able to override that.

Does all the above make sense ?

Even

> 
> -Jukka Rahkonen-
> 
> Even Rouault wrote:
> > Hi Steve,
> > 
> >> Hi Even: I take it that "the_normal_sql_query" could also include the
> >> translated filter described by a layers FILTER/FILTERITEM combo or the
> >> NATIVE_FILTER processing option, correct?
> > 
> > Hum, I was convinced that my implementation in the PostGIS driver already
> > did that, but it didn't ! And by further checking, I also saw I missed
> > that
> 
> msQueryByFilter() could be called indirectly by the WFS code through
> FLTxxxxx().
> 
> > Those 2 issues are now fixed with :
> https://github.com/rouault/mapserver/commit/4557afed33f92d22c969282d22dacf0
> 27e5e7387
> https://github.com/rouault/mapserver/commit/28354fe4bd5f5c3058a87ebc31d55c
> a7905388f3
> 
> > I've also added new test cases in msautotest:
> https://github.com/mapserver/msautotest/commit/05e8264ce737a795d81f1a87614f
> a5c87399de91
> 
> >> The only other gotcha is how (or if) to deal with class-level templates.
> >> The presence of a template is used to decide if a feature is to be
> >>
> >>processed as part of query operation and, in theory, if it should be
> >>
> >> counted. Typically one would configure a template at the LAYER-level
> > 
> >> but it's also possible to define CLASS-level templates. For example:
> > CLASSITEM 'column'
> > 
> >> CLASS
> >> 
> >>   EXRESSION 'foo'
> >>   TEMPLATE 'void'
> >>   ...
> >> 
> >> END
> >> CLASS
> >> 
> >>   EXRESSION 'bar'
> >>   # no template
> >>   ...
> >> 
> >> END
> >> 
> >> In this case you wouldn't know if you should count a feature until you
> >> know what CLASS it is. Granted this is an older configuration option
> >> that probably isn't widely used but it does exist. Perhaps it's
> >> currently broken anyway, I haven't tested. Just not sure how to deal
> >> with it. Perhaps it just needs to be noted as a configuration exception
> >> folks need to avoid when setting up WFS. I'd be happy to see that
> >> option removed in the next version - it adds unnecessary complexity to
> >> the query code. You can see the affect in msIsLayerQueryable() in
> >> mapquery.c  among other places in that source file.
> > 
> > I indeed see such stuff in msQueryByRect() and didn't know what it was
> > supposed to do. In fact, the WFS code has always forced a dummy
> > "ttt.html" layer template (if no layer template is defined), so class
> > templates are de-facto unused in WFS (since the msQueryXXXX() functions
> > do no test class templates if a layer template is defined). And the
> > msLayerGetShapeCount() optimization path is only taken if a layer
> > template is defined. So for non-WFS case, map-
> >
> >query.only_cache_result_count == 1 and class templates will still work
> >as
> 
> before.
> 
> > I've modified the RFC with the above changes and precisions :
> https://github.com/mapserver/docs/commit/76dfe8cb73c9c3de2f2e878e7c856dedc7
> a10c28
> 
> > Thanks a lot for your very much appreciated input !
> > 
> > Unrelated note: my latest commit fails in renderers/wmsclient.map because
> 
> http://demo.mapserver.org/cgi-
> bin/mapserv?map=/osgeo/mapserver/msautotest/world/world.map&service=WMS&req
> uest=GetCapabilities doesn't respond.
> 
> > Even
> > 
> >> Steve
> > 
> > -----Original Message-----
> > From: mapserver-dev [mailto:mapserver-dev-bounces at lists.osgeo.org] On
> > Behalf Of Even Rouault Sent: Monday, February 08, 2016 4:58 AM
> > To: MapServer Dev Mailing List <mapserver-dev at lists.osgeo.org>
> > Subject: [mapserver-dev] Call for discussion on MS RFC 114: Faster
> > retrieval of shape count
> > 
> > Hi,
> > 
> > Your comments on the following RFC are welcome :
> > http://mapserver.org/development/rfc/ms-rfc-114.html
> > 
> > Even
> 
> --
> Spatialys - Geospatial professional services http://www.spatialys.com
> _______________________________________________ mapserver-dev mailing list
> mapserver-dev at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/mapserver-dev

-- 
Spatialys - Geospatial professional services
http://www.spatialys.com