[gdal-dev] WFS requests failing with /vsicurl/ against QGIS Server: missing Content-Length header

Mon May 29 15:54:26 PDT 2023

Jorge,

Which error for which invocation of the WFS driver do you get ? Your 
below tries are already low level attempts.

e.g the below invocation works for me without any error with GDAL master 
(in "ghcr.io/osgeo/gdal:alpine-small-3.7.0" Docker image too)

ogr2ogr tmp.gpkg 
"WFS:https://irig-madeira.pt/postgresql/drote/public/baram_2020/cgi-bin/qgis_mapserv.fcgi?service=WFS"

The WFS driver doesn't use /vsicurl/ but /vsicurl_streaming/ . The 
former requires HTTP GET Range request support, while the later doesn't.

and when using "ogrinfo http://something", you are *not* using /vsicurl 
or /vsicurl_streaming. You are using the hidden HTTP driver that 
downloads locally the file before passing it to a driver.

I don't believe there is an issue with QGIS Server. It probably streams 
its output (didn't check though), so it cannot know in advance the 
Content-Length. The headers of the HEAD requests aren't particularly 
helpful too. Looking at the ones of the GET request, I see 
"Transfer-Encoding: chunked" for QGIS server, which is in line with a 
streamed output and OK

Even

Le 30/05/2023 à 00:23, Jorge Gustavo Rocha a écrit :
> Hi,
>
> I'm getting errors with WFS calls to QGIS Server using /vsicurl/. 
> Against Geoserver I didn't notice this issue.
>
> I can work on a solution, if the problem is on the QGIS Server side. I 
> would love to have a second opinion on this.
>
> What I have found:
>
> 1) The is no Content-Length: header in QGIS Server answer:
>
> curl -L -I 
> "https://irig-madeira.pt/postgresql/drote/public/baram_2020/cgi-bin/qgis_mapserv.fcgi?service=WFS&version=2.0.0&request=GetFeature&typename=ponto_cotado&srsname=EPSG:5016"
> HTTP/1.1 200 OK
> Date: Mon, 29 May 2023 22:12:32 GMT
> Server: Apache/2.4.41 (Ubuntu)
> Access-Control-Allow-Origin: *
> Access-Control-Allow-Methods: POST, GET, OPTIONS, DELETE, PUT
> Access-Control-Max-Age: 1000
> Access-Control-Allow-Headers: x-requested-with, Content-Type, origin, 
> authorization, accept, client-security-token
> Content-Type: text/xml; subtype=gml/2.1.2; charset=utf-8
>
> 2) The is a Content-Length: header in Geoserver answer:
>
> curl -L -I 
> "https://wssig3.azores.gov.pt/geoserver/ows?service=WFS&request=GetFeature&typename=ef:EnvironmentalMonitoringFacility&version=2.0.0&srsname=EPSG:5015"
> HTTP/1.1 200 OK
> X-Frame-Options: SAMEORIGIN
> Content-Type: application/gml+xml; version=3.2
> Content-Disposition: inline; filename=features.xml
> Content-Length: 494599
> Server: Jetty(9.4.18.v20190429)
>
> Example of testing calls:
>
> This works (without vsicurl):
> ogrinfo 
> "https://irig-madeira.pt/postgresql/drote/public/baram_2020/cgi-bin/qgis_mapserv.fcgi?service=WFS&version=2.0.0&request=GetFeature&typename=ponto_cotado&srsname=EPSG:5016"
> INFO: Open of 
> `https://irig-madeira.pt/postgresql/drote/public/baram_2020/cgi-bin/qgis_mapserv.fcgi?service=WFS&version=2.0.0&request=GetFeature&typename=ponto_cotado&srsname=EPSG:5016'
>       using driver `GML' successful.
> 1: ponto_cotado
>
> This does not work:
> ogrinfo 
> "/vsicurl/https://irig-madeira.pt/postgresql/drote/public/baram_2020/cgi-bin/qgis_mapserv.fcgi?service=WFS&version=2.0.0&request=GetFeature&typename=ponto_cotado&srsname=EPSG:5016" 
> --debug on
> HTTP: libcurl/7.81.0 OpenSSL/3.0.2 zlib/1.2.11 brotli/1.0.9 zstd/1.4.8 
> libidn2/2.3.2 libpsl/0.21.0 (+libidn2/2.3.2) libssh/0.9.6/openssl/zlib 
> nghttp2/1.43.0 librtmp/2.3 OpenLDAP/2.5.14
> VSICURL: HEAD did not provide file size. Retrying with GET
> VSICURL: 
> GetFileSize(https://irig-madeira.pt/postgresql/drote/public/baram_2020/cgi-bin/qgis_mapserv.fcgi?service=WFS&version=2.0.0&request=GetFeature&typename=ponto_cotado&srsname=EPSG:5016): 
> response_code=200, curl error msg=Failure writing output to destination
> VSICURL: Request at offset 0, after end of file
> VSICURL: Request at offset 0, after end of file
> VSICURL: 
> GetFileList(/vsicurl/https://irig-madeira.pt/postgresql/drote/public/baram_2020/cgi-bin)
> VSICURL: HEAD did not provide file size. Retrying with GET
> VSICURL: 
> GetFileSize(https://irig-madeira.pt/postgresql/drote/public/baram_2020/cgi-bin/qgis_mapserv.fcgi?service=WFS&version=2.0.0&request=GetFeature&typename=ponto_cotado&srsname=EPSG:5011): 
> response_code=200, curl error msg=Failure writing output to destination
> VSICURL: Request at offset 0, after end of file
> VSICURL: Request at offset 0, after end of file
> VSICURL: Request at offset 0, after end of file
> VSICURL: Request at offset 0, after end of file
> ogrinfo failed - unable to open 
> '/vsicurl/https://irig-madeira.pt/postgresql/drote/public/baram_2020/cgi-bin/qgis_mapserv.fcgi?service=WFS&version=2.0.0&request=GetFeature&typename=ponto_cotado&srsname=EPSG:5016'.
> DGNv8: Driver cleanup
>
> Using docker, I can not use /vsicurl/ with WFS too:
> docker run -ti --rm osgeo/gdal ogrinfo 
> "/vsicurl/https://irig-madeira.pt/postgresql/drote/public/baram_2020/cgi-bin/qgis_mapserv.fcgi?service=WFS&version=2.0.0&request=GetFeature&typename=ponto_cotado&srsname=EPSG:5016" 
> --debug on
> HTTP: libcurl/7.81.0 OpenSSL/3.0.2 zlib/1.2.11 brotli/1.0.9 zstd/1.4.8 
> libidn2/2.3.2 libpsl/0.21.0 (+libidn2/2.3.2) libssh/0.9.6/openssl/zlib 
> nghttp2/1.43.0 librtmp/2.3 OpenLDAP/2.5.13
> VSICURL: HEAD did not provide file size. Retrying with GET
> VSICURL: 
> GetFileSize(https://irig-madeira.pt/postgresql/drote/public/baram_2020/cgi-bin/qgis_mapserv.fcgi?service=WFS&version=2.0.0&request=GetFeature&typename=ponto_cotado&srsname=EPSG:5016): 
> response_code=200, curl error msg=Failure writing output to destination
> VSICURL: Request at offset 0, after end of file
> VSICURL: Request at offset 0, after end of file
> VSICURL: 
> GetFileList(/vsicurl/https://irig-madeira.pt/postgresql/drote/public/baram_2020/cgi-bin)
> VSICURL: HEAD did not provide file size. Retrying with GET
> VSICURL: 
> GetFileSize(https://irig-madeira.pt/postgresql/drote/public/baram_2020/cgi-bin/qgis_mapserv.fcgi?service=WFS&version=2.0.0&request=GetFeature&typename=ponto_cotado&srsname=EPSG:5011): 
> response_code=200, curl error msg=Failure writing output to destination
> VSICURL: Request at offset 0, after end of file
> VSICURL: Request at offset 0, after end of file
> VSICURL: Request at offset 0, after end of file
> VSICURL: Request at offset 0, after end of file
> ogrinfo failed - unable to open 
> '/vsicurl/https://irig-madeira.pt/postgresql/drote/public/baram_2020/cgi-bin/qgis_mapserv.fcgi?service=WFS&version=2.0.0&request=GetFeature&typename=ponto_cotado&srsname=EPSG:5016'.
>
> For a Geoserver layer, vsicurl is working:
>
> ogrinfo 
> "https://wssig3.azores.gov.pt/geoserver/ows?service=WFS&request=GetFeature&typename=ef:EnvironmentalMonitoringFacility&version=2.0.0&srsname=EPSG:5015"
> INFO: Open of 
> `https://wssig3.azores.gov.pt/geoserver/ows?service=WFS&request=GetFeature&typename=ef:EnvironmentalMonitoringFacility&version=2.0.0&srsname=EPSG:5015'
>       using driver `GML' successful.
> 1: EnvironmentalMonitoringFacility
>
> Can I assume that the problem is the missing Content-Length header in 
> QGIS Server WFS answer? Should I file a bug about this in QGIS?
>
> Regards,
>
> Jorge
>
-- 
http://www.spatialys.com
My software is free, but my time generally not.