[gdal-dev] WFS requests failing with /vsicurl/ against QGIS Server: missing Content-Length header

Mon May 29 16:14:08 PDT 2023

Hi Evan,

Thank you for the prompted answer.

I'm discussing this because I testing ogr2ogr node module and it adds 
(always) the /vsicurl/ prefix for any resource starting with http [1].
The point is /vsicurl/ can only be used with some http resources.

My question was (but you already answered it) if every WFS request 
should be callable with /vsicurl/.

Geoserver supports both /vsicurl/ and /vsicurl_streaming/ which seems 
more flexible. That's why I thought that QGIS Server should add 
Content-Length headers too.

If you consider that this is not a limitation of QGIS Server, let's keep 
as it is. I'll use /vsicurl_streaming/ for both Geoserver and QGIS 
Server calls.

Thanks for the feedback on this.

Best regards,

Jorge

[1] 
https://github.com/wavded/ogr2ogr/blob/729f321dd96414e256aa67be1bc76f0b0a9b21dc/index.ts#L105

On 29/05/23 23:54, Even Rouault wrote:
> Jorge,
> 
> Which error for which invocation of the WFS driver do you get ? Your 
> below tries are already low level attempts.
> 
> e.g the below invocation works for me without any error with GDAL master 
> (in "ghcr.io/osgeo/gdal:alpine-small-3.7.0" Docker image too)
> 
> ogr2ogr tmp.gpkg 
> "WFS:https://irig-madeira.pt/postgresql/drote/public/baram_2020/cgi-bin/qgis_mapserv.fcgi?service=WFS"
> 
> The WFS driver doesn't use /vsicurl/ but /vsicurl_streaming/ . The 
> former requires HTTP GET Range request support, while the later doesn't.
> 
> and when using "ogrinfo http://something", you are *not* using /vsicurl 
> or /vsicurl_streaming. You are using the hidden HTTP driver that 
> downloads locally the file before passing it to a driver.
> 
> I don't believe there is an issue with QGIS Server. It probably streams 
> its output (didn't check though), so it cannot know in advance the 
> Content-Length. The headers of the HEAD requests aren't particularly 
> helpful too. Looking at the ones of the GET request, I see 
> "Transfer-Encoding: chunked" for QGIS server, which is in line with a 
> streamed output and OK
> 
> Even
> 
> Le 30/05/2023 à 00:23, Jorge Gustavo Rocha a écrit :
>> Hi,
>>
>> I'm getting errors with WFS calls to QGIS Server using /vsicurl/. 
>> Against Geoserver I didn't notice this issue.
>>
>> I can work on a solution, if the problem is on the QGIS Server side. I 
>> would love to have a second opinion on this.
>>
>> What I have found:
>>
>> 1) The is no Content-Length: header in QGIS Server answer:
>>
>> curl -L -I 
>> "https://irig-madeira.pt/postgresql/drote/public/baram_2020/cgi-bin/qgis_mapserv.fcgi?service=WFS&version=2.0.0&request=GetFeature&typename=ponto_cotado&srsname=EPSG:5016"
>> HTTP/1.1 200 OK
>> Date: Mon, 29 May 2023 22:12:32 GMT
>> Server: Apache/2.4.41 (Ubuntu)
>> Access-Control-Allow-Origin: *
>> Access-Control-Allow-Methods: POST, GET, OPTIONS, DELETE, PUT
>> Access-Control-Max-Age: 1000
>> Access-Control-Allow-Headers: x-requested-with, Content-Type, origin, 
>> authorization, accept, client-security-token
>> Content-Type: text/xml; subtype=gml/2.1.2; charset=utf-8
>>
>> 2) The is a Content-Length: header in Geoserver answer:
>>
>> curl -L -I 
>> "https://wssig3.azores.gov.pt/geoserver/ows?service=WFS&request=GetFeature&typename=ef:EnvironmentalMonitoringFacility&version=2.0.0&srsname=EPSG:5015"
>> HTTP/1.1 200 OK
>> X-Frame-Options: SAMEORIGIN
>> Content-Type: application/gml+xml; version=3.2
>> Content-Disposition: inline; filename=features.xml
>> Content-Length: 494599
>> Server: Jetty(9.4.18.v20190429)
>>
>> Example of testing calls:
>>
>> This works (without vsicurl):
>> ogrinfo 
>> "https://irig-madeira.pt/postgresql/drote/public/baram_2020/cgi-bin/qgis_mapserv.fcgi?service=WFS&version=2.0.0&request=GetFeature&typename=ponto_cotado&srsname=EPSG:5016"
>> INFO: Open of 
>> `https://irig-madeira.pt/postgresql/drote/public/baram_2020/cgi-bin/qgis_mapserv.fcgi?service=WFS&version=2.0.0&request=GetFeature&typename=ponto_cotado&srsname=EPSG:5016'
>>       using driver `GML' successful.
>> 1: ponto_cotado
>>
>> This does not work:
>> ogrinfo 
>> "/vsicurl/https://irig-madeira.pt/postgresql/drote/public/baram_2020/cgi-bin/qgis_mapserv.fcgi?service=WFS&version=2.0.0&request=GetFeature&typename=ponto_cotado&srsname=EPSG:5016" --debug on
>> HTTP: libcurl/7.81.0 OpenSSL/3.0.2 zlib/1.2.11 brotli/1.0.9 zstd/1.4.8 
>> libidn2/2.3.2 libpsl/0.21.0 (+libidn2/2.3.2) libssh/0.9.6/openssl/zlib 
>> nghttp2/1.43.0 librtmp/2.3 OpenLDAP/2.5.14
>> VSICURL: HEAD did not provide file size. Retrying with GET
>> VSICURL: 
>> GetFileSize(https://irig-madeira.pt/postgresql/drote/public/baram_2020/cgi-bin/qgis_mapserv.fcgi?service=WFS&version=2.0.0&request=GetFeature&typename=ponto_cotado&srsname=EPSG:5016): response_code=200, curl error msg=Failure writing output to destination
>> VSICURL: Request at offset 0, after end of file
>> VSICURL: Request at offset 0, after end of file
>> VSICURL: 
>> GetFileList(/vsicurl/https://irig-madeira.pt/postgresql/drote/public/baram_2020/cgi-bin)
>> VSICURL: HEAD did not provide file size. Retrying with GET
>> VSICURL: 
>> GetFileSize(https://irig-madeira.pt/postgresql/drote/public/baram_2020/cgi-bin/qgis_mapserv.fcgi?service=WFS&version=2.0.0&request=GetFeature&typename=ponto_cotado&srsname=EPSG:5011): response_code=200, curl error msg=Failure writing output to destination
>> VSICURL: Request at offset 0, after end of file
>> VSICURL: Request at offset 0, after end of file
>> VSICURL: Request at offset 0, after end of file
>> VSICURL: Request at offset 0, after end of file
>> ogrinfo failed - unable to open 
>> '/vsicurl/https://irig-madeira.pt/postgresql/drote/public/baram_2020/cgi-bin/qgis_mapserv.fcgi?service=WFS&version=2.0.0&request=GetFeature&typename=ponto_cotado&srsname=EPSG:5016'.
>> DGNv8: Driver cleanup
>>
>> Using docker, I can not use /vsicurl/ with WFS too:
>> docker run -ti --rm osgeo/gdal ogrinfo 
>> "/vsicurl/https://irig-madeira.pt/postgresql/drote/public/baram_2020/cgi-bin/qgis_mapserv.fcgi?service=WFS&version=2.0.0&request=GetFeature&typename=ponto_cotado&srsname=EPSG:5016" --debug on
>> HTTP: libcurl/7.81.0 OpenSSL/3.0.2 zlib/1.2.11 brotli/1.0.9 zstd/1.4.8 
>> libidn2/2.3.2 libpsl/0.21.0 (+libidn2/2.3.2) libssh/0.9.6/openssl/zlib 
>> nghttp2/1.43.0 librtmp/2.3 OpenLDAP/2.5.13
>> VSICURL: HEAD did not provide file size. Retrying with GET
>> VSICURL: 
>> GetFileSize(https://irig-madeira.pt/postgresql/drote/public/baram_2020/cgi-bin/qgis_mapserv.fcgi?service=WFS&version=2.0.0&request=GetFeature&typename=ponto_cotado&srsname=EPSG:5016): response_code=200, curl error msg=Failure writing output to destination
>> VSICURL: Request at offset 0, after end of file
>> VSICURL: Request at offset 0, after end of file
>> VSICURL: 
>> GetFileList(/vsicurl/https://irig-madeira.pt/postgresql/drote/public/baram_2020/cgi-bin)
>> VSICURL: HEAD did not provide file size. Retrying with GET
>> VSICURL: 
>> GetFileSize(https://irig-madeira.pt/postgresql/drote/public/baram_2020/cgi-bin/qgis_mapserv.fcgi?service=WFS&version=2.0.0&request=GetFeature&typename=ponto_cotado&srsname=EPSG:5011): response_code=200, curl error msg=Failure writing output to destination
>> VSICURL: Request at offset 0, after end of file
>> VSICURL: Request at offset 0, after end of file
>> VSICURL: Request at offset 0, after end of file
>> VSICURL: Request at offset 0, after end of file
>> ogrinfo failed - unable to open 
>> '/vsicurl/https://irig-madeira.pt/postgresql/drote/public/baram_2020/cgi-bin/qgis_mapserv.fcgi?service=WFS&version=2.0.0&request=GetFeature&typename=ponto_cotado&srsname=EPSG:5016'.
>>
>> For a Geoserver layer, vsicurl is working:
>>
>> ogrinfo 
>> "https://wssig3.azores.gov.pt/geoserver/ows?service=WFS&request=GetFeature&typename=ef:EnvironmentalMonitoringFacility&version=2.0.0&srsname=EPSG:5015"
>> INFO: Open of 
>> `https://wssig3.azores.gov.pt/geoserver/ows?service=WFS&request=GetFeature&typename=ef:EnvironmentalMonitoringFacility&version=2.0.0&srsname=EPSG:5015'
>>       using driver `GML' successful.
>> 1: EnvironmentalMonitoringFacility
>>
>> Can I assume that the problem is the missing Content-Length header in 
>> QGIS Server WFS answer? Should I file a bug about this in QGIS?
>>
>> Regards,
>>
>> Jorge
>>

-- 
OSGeo Portugal
NIF: 509588190
email: geral at osgeopt.pt
https://www.osgeopt.pt/

Próximo evento: SASIG 2017, 20 a 22 de novembro, Porto
Mais informações: http://osgeopt.pt/sasig2017/