[gdal-dev] Deciding whether to use WFS paging after opening the datasource

Tue May 2 18:10:53 PDT 2023

Hi folks

We're having trouble with OGR trying to fetch a response from this WFS2
URL:
https://geo.irceline.be/wfs?SERVICE=WFS&VERSION=2.0.0&REQUEST=GetFeature&TYPENAMES=realtime:pm25_24hmean_station&COUNT=1000000&STARTINDEX=0

The error from the server is

Cannot do natural order without a primary key, please add it or specify a
> manual sort over existing attributes
>

This seems fair enough on the surface - we specified *STARTINDEX=0* but the
server is unable to support paging for this dataset due to no natural sort
order.

We'd happily disable paging for this dataset - it has a fairly low number
of features - but we would prefer to implement a general purpose solution
rather than special-case this particular dataset.

It would make sense to open the datasource, check the feature count by
doing a 'hits' query (we're doing this anyway), and then use paging if
there are more than 1000000 features - otherwise don't use paging.

However I can't find a way to accomplish this with GDAL. The
*OGR_WFS_PAGING_ALLOWED* config var is only checked when the datasource is
opened - so you can't open the datasource, check the feature count and then
change the paging behaviour.

I can imagine either of these changes in GDAL itself might fix this:

1. GDAL could check *OGR_WFS_PAGING_ALLOWED* when actually first issuing
GetFeature requests, rather than during datasource creation
2. GDAL could omit the pagination parameters if the feature count is
already known and is less than the page size

How would you suggest we proceed?

-- 
Regards,
Craig

Platform Engineer
Koordinates

+64 21 256 9488 <+64%2021%20256%209488> / koordinates.com / @koordinates
<https://twitter.com/koordinates>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20230503/184bc866/attachment.htm>