[gdal-dev] Deciding whether to use WFS paging after opening the datasource

[Even Rouault]

Craig,

Both options you propose are reasonable. I'd perhaps have a slight 
preference for option 2 (is my understanding correct that the user would 
be responsible to issue GetFeatureCount() manually before calling 
GetNextFeature() ? we don't want to systematically call 
GetFeatureCount() because it could be potentially a rather costly 
operation for some servers).

It could be enhanced with allowing a CHECK_WITH_HITS value for 
OGR_WFS_PAGING_ALLOWED (or perhaps through a new specific open option ?) 
that would automatically issue the feature count request (that could be 
useful for ogr2ogr types of scenario where the user can't easily cause 
ogr2ogr to issue GetFeatureCount())

Even

Le 03/05/2023 à 03:10, Craig de Stigter a écrit :
> Hi folks
>
> We're having trouble with OGR trying to fetch a response from this 
> WFS2 URL: 
> https://geo.irceline.be/wfs?SERVICE=WFS&VERSION=2.0.0&REQUEST=GetFeature&TYPENAMES=realtime:pm25_24hmean_station&COUNT=1000000&STARTINDEX=0 
> <https://geo.irceline.be/wfs?SERVICE=WFS&VERSION=2.0.0&REQUEST=GetFeature&TYPENAMES=realtime:pm25_24hmean_station&COUNT=1000000&STARTINDEX=0>
>
> The error from the server is
>
>     Cannot do natural order without a primary key, please add it or
>     specify a manual sort over existing attributes
>
>
> This seems fair enough on the surface - we specified *STARTINDEX=0* 
> but the server is unable to support paging for this dataset due to no 
> natural sort order.
>
> We'd happily disable paging for this dataset - it has a fairly low 
> number of features - but we would prefer to implement a general 
> purpose solution rather than special-case this particular dataset.
>
> It would make sense to open the datasource, check the feature count by 
> doing a 'hits' query (we're doing this anyway), and then use paging if 
> there are more than 1000000 features - otherwise don't use paging.
>
> However I can't find a way to accomplish this with GDAL. The 
> *OGR_WFS_PAGING_ALLOWED* config var is only checked when the 
> datasource is opened - so you can't open the datasource, check the 
> feature count and then change the paging behaviour.
>
> I can imagine either of these changes in GDAL itself might fix this:
>
> 1. GDAL could check *OGR_WFS_PAGING_ALLOWED* when actually first 
> issuing GetFeature requests, rather than during datasource creation
> 2. GDAL could omit the pagination parameters if the feature count is 
> already known and is less than the page size
>
> How would you suggest we proceed?
>
> -- 
> Regards,
> Craig
>
> Platform Engineer
> Koordinates
>
> +64 21 256 9488 <tel:+64%2021%20256%209488> / koordinates.com 
> <http://koordinates.com/> / @koordinates <https://twitter.com/koordinates>
>
> _______________________________________________
> gdal-dev mailing list
> gdal-dev at lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/gdal-dev

-- 
http://www.spatialys.com
My software is free, but my time generally not.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20230503/e3a96647/attachment.htm>