[gdal-dev] vsicurl configuration design decisions

[Even Rouault]

Hi Sean,

> 
> It's written in
> http://gdal.org/gdal_virtual_file_systems.html#gdal_virtual_file_systems_vsi
> curl
> > Starting with GDAL 2.3, options can be passed in the filename with the
> 
> following syntax: /vsicurl/option1=val1[,optionN=valN]*,url=http://...
> 
> I'd like to discuss the design decisions that are being made here before
> this gets out into the world.
> 
> I'm uncomfortable with the way configuration is spread between environment
> variables, config options that surface in the API,

Just a precision: GDAL only reads configuration options with CPLGetConfigOption(key). 
Those can be implictly set through environment variables of the same name or with 
CPLSetConfigOption(key, value).

> and also in identifiers.
> I don't think it's a great idea to that expand the amount of configuration
> in dataset identifiers. It's redundant, the syntax is complicated, 

Frank answered on the main motivations.

> and it
> dilutes the network effects of reusing identifiers in our applications.

Didn't understand what you meant with the above sentence.

> 
> Are there specific advantages to this
> 
>   ogrinfo -so /vsicurl/max_retry=10,url=https://example.com/poly.shp
> 
> that we can't also have with a curl-style
> 
>   ogrinfo -so --max-retry=10 /vsicurl/https://example.com/poly.shp
> 
> or, better yet, in my opinion
> 
>   ogrinfo -so --max-retry=10 https://example.com/poly.shp
> 
> on the command line?

One issue with you proposal is that it would require ogrinfo (or any utility) to go from the 
highest level abstraction layers of GDAL to the lowest ones.

When ogrinfo is provided
"/vsicurl/max_retry=10,url=https://example.com/poly.shp",
this is just a string used as a dataset name

It happily feeds it into GDALOpenEx(), which in turns proposes it sequentially to all drivers

The shapefile driver tries this string with VSIFOpenL(), which in turns iterates over all virtual 
file systems. The /vsicurl/ VFS happens to recognize it, manages to open the file. The 
shapefile driver can read the few first bytes from it and recognizes that it is a header of a 
shapefile, etc..

So in the current design neither the utility, nor GDALOpenEx(), or the drivers themselves 
really make a sense of that string. This is quite a strength at the architectural level. This also 
enables to pass such a string in a VRT file for example.

Regarding the direct use of http:// https:// , I also find it is a bit unfortunate that we can't use 
them directly and vsicurl machinery would be implictly used. It turns that historically we have 
the HTTP driver that triggers on such dataset name (ingesting the whole file into /vsimem/, 
and proposing it in turn to other drivers). There's also a few other drivers (DODS, etc..) that 
trigger on such names.

Even


-- 
Spatialys - Geospatial professional services
http://www.spatialys.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20171010/e11c141e/attachment.html>