[gdal-dev] vsicurl configuration design decisions

[Craig de Stigter]

Hi folks

We're slightly invested in this because we use VSI paths reasonably
heavily, though not so much for cloud services yet.


> One downside is that you need to URLEncode the URL, which can make it
> painful when composing it at hand.


True, but that does eliminate ambiguity in the URL, and does so in a
well-known way.

Does the current scheme use any encoding? How would you escape text in
option values that might use `=` and `,` etc? Or are there guaranteed to be
no freeform-text options in these paths?



Tangential, but related: I've also just discovered the 2.2+ curly-brace
syntax for vsizip/vsitar paths, which allows nested archives:

/vsizip/{/vsizip/{/path/to/outer.zip}/path/to/inner.zip}/file.shp


The curly braces are a definite improvement on the ambiguous older syntax
for these paths. However, we noted the nesting order looks inside-out, and
thought it would have been more intuitive to put the path *inside* the
archive in the braces. i.e. nesting would look like:

> /vsizip//path/to/outer.zip/{/vsizip//path/to/inner.zip/{file.shp}}


Of course, this latter syntax was added in 2.2, so perhaps that ship has
already sailed.

>From our experiences with vsicurl and vsizip urls, it feels like
eliminating ambiguity in these paths is pretty important, more so than
trivial composability. Just my 2¢ :)


On 13 October 2017 at 07:42, Even Rouault <even.rouault at spatialys.com>
wrote:

> Hi Sean,
>
>
>
> > Is the future of open and creation options?
>
>
>
> I don't understand your above sentence.
>
>
>
> > Do you imagine this extended
>
> > to, say, block size, compression, number of threads? An RFC that
> discussed
>
> > the scope of this and at what level of abstraction it is implemented at
>
> > might be warranted? I'd be happy to participate.
>
>
>
> Not clear what you've in mind. Are you thinking about some formalism to
> define and specify options for VSI filenames ?
>
>
>
> > On the other hand, https://example.com/foo.tif identifies only a single
>
> > resource, whereas /viscurl/url=https://example.com/foo.tif can identify
> a
>
> > GeoTIFF along with all of its sidecars. I presume that the new GDAL cloud
>
> > utilities like gdal_cp.py take care of the auxiliary files, yes?
>
>
>
> No. They should perhaps be named cpl_xxx since they really operate at the
> VSI/file level. Auxiliary/sidecar files are concepts that exist only at the
> driver level/
>
>
>
> Copy of datasets + side car files can be done with "gdalmanage copy"
>
>
>
> >
>
> > My final concern about the virtual file opening options is the syntax.
>
> > These /vsicurl/option1=val1[,optionN=valN]*,url=http://
> example.com/foo.tif
>
> > identifiers (or filenames or whatever we call them) may spread from GDAL
>
> > into the wider geospatial programming domain. Speaking from my experience
>
> > with Rasterio, open source Python GIS developers expect the /vsi*
> filenames
>
> > to "just work" in all software. Can we consider using a more standard
>
> > syntax? One that has parsers already deployed everywhere?
>
>
>
> I don't really see a use of parsing those /vsi names by user code. User
> code has to compose them, not parse them. But I can see your point for
> something more standardized.
>
>
>
> > That syntax gives the /vsi* filenames the form of a "reflector" URL such
> as
>
> > we see in Google searches (for example:
>
> > https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&
> cd=1&ved=0ahUKEwj
>
> > C6e7hvevWAhXmjFQKHWsHDyMQFggmMAA&url=http%3A%2F%2Fwww.gdal.org
> %2F&usg=AOvVaw
>
> > 3fbRv5TusYwkXgz2Acf2kt) and there are abundant tools and a body of
> knowledge
>
> > about how to parse and work with these.
>
>
>
> One downside is that you need to URLEncode the URL, which can make it
> painful when composing it at hand.
>
>
>
>
>
> --
>
> Spatialys - Geospatial professional services
>
> http://www.spatialys.com
>
> _______________________________________________
> gdal-dev mailing list
> gdal-dev at lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/gdal-dev
>



-- 
Regards,
Craig

Developer
Koordinates

+64 21 256 9488 <+64%2021%20256%209488> / koordinates.com / @koordinates
<https://twitter.com/koordinates>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20171013/3f7f49ed/attachment-0001.html>