[gdal-dev] OGR fails to open a VRT layer with a remote data source if VSI_CACHE=TRUE

[Andrea Giudiceandrea]

Hi all,
searching for the source of a QGIS issue
https://github.com/qgis/QGIS/issues/35016, after testing various QGIS and
GDAL version on Windows and Ubuntu Linux, I found that OGR fails to open a
VRT layer file, with a remote data source (a csv file) hosted on data.world
server, if VSI_CACHE is set to TRUE.

The QGIS issue was spotted on Windows since the OSGeo4W QGIS installer sets
by default VSI_CACHE=TRUE and VSI_CACHE_SIZE=1000000: it seems these
settings were introduced 7 years ago [1] to workaround a reported OGR
performance issue [2] about shapefile reading over network on Windows.

A test VRT file is available at
https://raw.githubusercontent.com/agiudiceandrea/TEST_CSV/master/covid19-regioni_dw.vrt

On both Windows (7 / 10 with GDAL/OGR 3.0.4 - 3.2.0dev) and Ubuntu Linux
(20.04 / 18.04 with GDAL/OGR 3.0.4 - 2.2.3)

if VSI_CACHE is not set or set to FALSE

ogrinfo covid19-regioni_dw.vrt jgsghstpphjhicstradhy5kpjwrnfy -summary

outputs

INFO: Open of `covid19-regioni_dw.vrt'
      using driver `OGR_VRT' successful.
Warning 6: Unknown type : data
...
Layer name: jgsghstpphjhicstradhy5kpjwrnfy
Geometry: Point
Feature Count: 30069
Extent: (0.000000, 0.000000) - (18.171897, 46.499335)
Layer SRS WKT:
GEOGCRS["WGS 84",
    DATUM["World Geodetic System 1984",
        ELLIPSOID["WGS 84",6378137,298.257223563,
            LENGTHUNIT["metre",1]]],
    PRIMEM["Greenwich",0,
        ANGLEUNIT["degree",0.0174532925199433]],
    CS[ellipsoidal,2],
        AXIS["geodetic latitude (Lat)",north,
            ORDER[1],
            ANGLEUNIT["degree",0.0174532925199433]],
        AXIS["geodetic longitude (Lon)",east,
            ORDER[2],
            ANGLEUNIT["degree",0.0174532925199433]],
    ID["EPSG",4326]]
Data axis to CRS axis mapping: 2,1
denominazione_regione: String (0.0)

and QGIS can properly display the layer

while if VSI_CACHE is set to TRUE

ogrinfo covid19-regioni_dw.vrt jgsghstpphjhicstradhy5kpjwrnfy -summary

outputs

INFO: Open of `covid19-regioni_dw.vrt'
      using driver `OGR_VRT' successful.
ERROR 1: Failed to open datasource
`CSV:/vsicurl_streaming/https://query.data.world/s/jgsghstpphjhicstradhy5kpjwrnfy'.

Layer name: jgsghstpphjhicstradhy5kpjwrnfy
Geometry: None
Feature Count: 0
Layer SRS WKT:
(unknown)

and QGIS cannot display the layer.


Could the issue be caused by a server misconfiguration or not suitable for
this kind of use, or should it be fixed in GDAL/OGR?

Do you think the VSI_CACHE=TRUE 7 years old workaround setting is really
still required on Windows (OSGeo4W), since in the meantime GDAL/OGR may have
been improved in shapefile reading over network?


If it could help, I can provide further analysis on HTTPS traffic and QGIS
debug log during the connection attempts.


Best regards.

Andrea Giudiceandrea


[1]
https://github.com/qgis/QGIS/commit/9222f152b30948a3a9d9f66dffd162548022f7fd

[2]
https://lists.osgeo.org/pipermail/gdal-dev/2013-April/036077.html
https://issues.qgis.org/issues/6448 |
https://github.com/qgis/QGIS/issues/15688



--
Sent from: http://osgeo-org.1560.x6.nabble.com/GDAL-Dev-f3742093.html