[mapserver-dev] Call for discussion on MS RFC 114: Faster retrieval of shape count

Wed Feb 10 04:51:15 PST 2016

Le mercredi 10 février 2016 13:34:00, Rahkonen Jukka (MML) a écrit :
> Hi,
> 
> I mean both server side and client side maxFeatures settings. As I read the
> standard the &resultType=hits should return a "numberOfFeatures" that is
> equal to the number of the features which are really returned if user
> makes the request as &resultType=results. But if implemented that way
> users have no means to check the total count of features belonging to the
> featureType if that count is bigger than the server side maxFeatures
> limit. So the "wfs_ignore_server_maxfeatures_for_hits" would be useful to
> have but I believe that it is not in the WFS standard. I may be wrong,
> this would not be the first time.

Looking in the comments of mapwfs.c I wrote during WFS 2.0 implementation, I 
see "Deegree (http://deegree3-demo.deegree.org:80/inspire-
workspace/services/wfs?) will return numberMatched="19005" on the 
cp:CadastralParcel layer for a  resultType=hits request, whereas the server 
limitation is set to 15000.GeoServer 2.4.1 will also ignore its server 
limitation for a resultType=hits"

And just tried :
http://demo.opengeo.org/geoserver/wfs?SERVICE=WFS&VERSION=2.0.0&REQUEST=GetFeature&TYPENAMES=osm:minor_roads_casing&RESULTTYPE=hits

--> returns numberMatched="1494887"

whereas 
http://demo.opengeo.org/geoserver/wfs?SERVICE=WFS&VERSION=2.0.0&REQUEST=GetCapabilities
mentions CountDefault = 1000000

But I agree with you that the text of the standard could make think that the 
server side limit should be taken into account by resultType=hits, although it 
is not explicit at all...

> 
> -Jukka-
> 
> Even Rouault wrote:
> > Hi Jukka,
> > 
> > 
> > I am thinking about the metadata setting
> > “wfs_ignore_maxfeatures_for_hits”. While it is certainly useful for the
> > users sometimes I believe that ignoring maxfeatures is still not WFS
> > standard compliant.
> 
> Perhaps there is an ambiguity in the name since MAXFEATURES is also a WFS
> 1.1 parameter. "wfs_ignore_maxfeatures_for_hits" is aimed at ignoring the
> *server* "wfs_maxfeatures" setting, not the client one.
> 
> So perhaps "wfs_ignore_server_maxfeatures_for_hits" ?
> 
> Client MAXFEATURES will always be honoured (if no greater than server
> "wfs_maxfeatures")
> 
> > Quotation from
> > WFS 1.1.0 standard:
> > 
> > "In this way  [by using resultType=hits] a client may obtain a count
> > of the number of features that a query would return without having to
> > incur the cost of transmitting the entire result set."
> 
> MapServer current behaviour was non compliant in case a "wfs_maxfeatures"
> was defined in the mapfile. With this RFC, in the PostGIS case, it will
> now be compliant by default. We could default
> "wfs_ignore_maxfeatures_for_hits" to "true" for other providers as well,
> but there might be a performance hit : imagine querying a huge shapefile,
> OGR datasource (and also for now Oracle, MSSQL) with > millions of
> records. This would mean all those records would be retrieved by MapServer
> to count them. Administrators might want to be able to avoid that.
> 
> > If ignore maxfeatures is set in the metadata it makes that WFS service
> > permanently non-compliant.
> 
> non-compliant if it is set to "false".
> 
> This RFC makes MapServer compliant for PostGIS by default. And for other
> backends if "wfs_ignore_maxfeatures_for_hits" is explicitly set to "true".
> That seems a good compromise to me.
> 
> > I wonder if we could have a vendor parameter like
> > "&feature_count=total" for controlling which count &resultType=hits
> > will return.  For end user both counts can be valuable: total count
> > for knowing the size of the layer and the count that takes maxFeatures
> > into account the maximum size that can be used for paging.
> 
> - "total count for knowing the size of the layer":  that's what you get if
> "wfs_ignore_maxfeatures_for_hits" is set to "true" (explictly or
> implictly) - "the count that takes maxFeatures into account the maximum
> size that can be used for paging" : given the above total count and the
> WFS 1.1 "DefaultMaxFeatures" / WFS 2.0 "CountDefault" values returned in
> GetCapabilities response, you can know if paging will be needed.
> 
> I don't think we need & want a vendor parameter (there's already enough
> complexity with the combinations of client MAXFEATURES parameter and
> server side limitations I think). The purpose of
> "wfs_ignore_server_maxfeatures_for_hits" and "wfs_maxfeatures" is to
> control how much we want a user request to be able to make the data
> backend busy, sometimes at the expense of strict standard conformance.
> Users should not be able to override that.
> 
> Does all the above make sense ?
> 
> Even
> 
> > -Jukka Rahkonen-
> > 
> > Even Rouault wrote:
> > > Hi Steve,
> > > 
> > >> Hi Even: I take it that "the_normal_sql_query" could also include
> > >> the translated filter described by a layers FILTER/FILTERITEM combo
> > >> or the NATIVE_FILTER processing option, correct?
> > > 
> > > Hum, I was convinced that my implementation in the PostGIS driver
> > > already did that, but it didn't ! And by further checking, I also
> > > saw I missed that
> > 
> > msQueryByFilter() could be called indirectly by the WFS code through
> > FLTxxxxx().
> > 
> > > Those 2 issues are now fixed with :
> > https://github.com/rouault/mapserver/commit/4557afed33f92d22c969282d22
> > dacf0
> > 27e5e7387
> > https://github.com/rouault/mapserver/commit/28354fe4bd5f5c3058a87ebc31
> > d55c
> > a7905388f3
> > 
> > > I've also added new test cases in msautotest:
> > https://github.com/mapserver/msautotest/commit/05e8264ce737a795d81f1a8
> > 7614f
> > a5c87399de91
> > 
> > >> The only other gotcha is how (or if) to deal with class-level
> > >> templates. The presence of a template is used to decide if a feature
> > >> is to be
> > >>
> > >>processed as part of query operation and, in theory, if it should be
> > >>
> > >> counted. Typically one would configure a template at the
> > >> LAYER-level
> > > 
> > >> but it's also possible to define CLASS-level templates. For example:
> > > CLASSITEM 'column'
> > > 
> > >> CLASS
> > >> 
> > >>   EXRESSION 'foo'
> > >>   TEMPLATE 'void'
> > >>   ...
> > >> 
> > >> END
> > >> CLASS
> > >> 
> > >>   EXRESSION 'bar'
> > >>   # no template
> > >>   ...
> > >> 
> > >> END
> > >> 
> > >> In this case you wouldn't know if you should count a feature until
> > >> you know what CLASS it is. Granted this is an older configuration
> > >> option that probably isn't widely used but it does exist. Perhaps
> > >> it's currently broken anyway, I haven't tested. Just not sure how
> > >> to deal with it. Perhaps it just needs to be noted as a
> > >> configuration exception folks need to avoid when setting up WFS.
> > >> I'd be happy to see that option removed in the next version - it
> > >> adds unnecessary complexity to the query code. You can see the
> > >> affect in msIsLayerQueryable() in mapquery.c  among other places in
> > >> that source file.
> > > 
> > > I indeed see such stuff in msQueryByRect() and didn't know what it
> > > was supposed to do. In fact, the WFS code has always forced a dummy
> > > "ttt.html" layer template (if no layer template is defined), so
> > > class templates are de-facto unused in WFS (since the msQueryXXXX()
> > > functions do no test class templates if a layer template is
> > > defined). And the
> > > msLayerGetShapeCount() optimization path is only taken if a layer
> > > template is defined. So for non-WFS case, map-
> > >
> > >query.only_cache_result_count == 1 and class templates will still
> > >work as
> > 
> > before.
> > 
> > > I've modified the RFC with the above changes and precisions :
> > https://github.com/mapserver/docs/commit/76dfe8cb73c9c3de2f2e878e7c856
> > dedc7
> > a10c28
> > 
> > > Thanks a lot for your very much appreciated input !
> > > 
> > > Unrelated note: my latest commit fails in renderers/wmsclient.map
> > > because
> > 
> > http://demo.mapserver.org/cgi-
> > bin/mapserv?map=/osgeo/mapserver/msautotest/world/world.map&service=WM
> > S&req
> > uest=GetCapabilities doesn't respond.
> > 
> > > Even
> > > 
> > >> Steve
> > > 
> > > -----Original Message-----
> > > From: mapserver-dev [mailto:mapserver-dev-bounces at lists.osgeo.org]
> > > On Behalf Of Even Rouault Sent: Monday, February 08, 2016 4:58 AM
> > > To: MapServer Dev Mailing List <mapserver-dev at lists.osgeo.org>
> > > Subject: [mapserver-dev] Call for discussion on MS RFC 114: Faster
> > > retrieval of shape count
> > > 
> > > Hi,
> > > 
> > > Your comments on the following RFC are welcome :
> > > http://mapserver.org/development/rfc/ms-rfc-114.html
> > > 
> > > Even
> > 
> > --
> > Spatialys - Geospatial professional services http://www.spatialys.com
> > _______________________________________________ mapserver-dev mailing
> > list mapserver-dev at lists.osgeo.org
> > http://lists.osgeo.org/mailman/listinfo/mapserver-dev
> 
> --
> Spatialys - Geospatial professional services http://www.spatialys.com

-- 
Spatialys - Geospatial professional services
http://www.spatialys.com