<div dir="ltr"><div>Hi folks</div><div><br></div><div>We're slightly invested in this because we use VSI paths reasonably heavily, though not so much for cloud services yet.</div><div> </div><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote">One downside is that you need to URLEncode the URL, which can make it painful when composing it at hand.</blockquote><div><br></div><div>True, but that does eliminate ambiguity in the URL, and does so in a well-known way.</div><div><br></div><div>Does the current scheme use any encoding? How would you escape text in option values that might use `=` and `,` etc? Or are there guaranteed to be no freeform-text options in these paths?</div><div><br></div><div><br></div><div><br></div><div>Tangential, but related: I've also just discovered the 2.2+ curly-brace syntax for vsizip/vsitar paths, which allows nested archives:<br></div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><font face="monospace, monospace">/vsizip/{/vsizip/{/path/to/outer.zip}/path/to/inner.zip}/file.shp</font></blockquote><div><br></div><div>The curly braces are a definite improvement on the ambiguous older syntax for these paths. However, we noted the nesting order looks inside-out, and thought it would have been more intuitive to put the path <i>inside</i> the archive in the braces. i.e. nesting would look like:</div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><font face="monospace, monospace">/vsizip//path/to/outer.zip/{/vsizip//path/to/inner.zip/{file.shp}}</font></blockquote><div><br></div><div>Of course, this latter syntax was added in 2.2, so perhaps that ship has already sailed.</div><div><br></div><div>From our experiences with vsicurl and vsizip urls, it feels like eliminating ambiguity in these paths is pretty important, more so than trivial composability. Just my 2¢ :)</div><div><br></div><span class="gmail-im"><span style="font-family:monospace;font-size:12px"></span></span><div class="gmail_extra"><br><div class="gmail_quote">On 13 October 2017 at 07:42, Even Rouault <span dir="ltr"><<a href="mailto:even.rouault@spatialys.com" target="_blank">even.rouault@spatialys.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><u></u>
<div style="font-family:'monospace';font-size:9pt;font-weight:400;font-style:normal">
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">Hi Sean,</p><span class="">
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px"> </p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> Is the future of open and creation options?</p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px"> </p>
</span><p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">I don't understand your above sentence.</p><span class="">
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px"> </p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> Do you imagine this extended</p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> to, say, block size, compression, number of threads? An RFC that discussed</p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> the scope of this and at what level of abstraction it is implemented at</p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> might be warranted? I'd be happy to participate.</p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px"> </p>
</span><p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">Not clear what you've in mind. Are you thinking about some formalism to define and specify options for VSI filenames ?</p><span class="">
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px"> </p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> On the other hand, <a href="https://example.com/foo.tif" target="_blank">https://example.com/foo.tif</a> identifies only a single</p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> resource, whereas /viscurl/url=<a href="https://example.com/foo.tif" target="_blank">https://example.<wbr>com/foo.tif</a> can identify a</p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> GeoTIFF along with all of its sidecars. I presume that the new GDAL cloud</p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> utilities like gdal_cp.py take care of the auxiliary files, yes?</p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px"> </p>
</span><p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">No. They should perhaps be named cpl_xxx since they really operate at the VSI/file level. Auxiliary/sidecar files are concepts that exist only at the driver level/</p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px"> </p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">Copy of datasets + side car files can be done with "gdalmanage copy"</p><span class="">
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px"> </p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> </p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> My final concern about the virtual file opening options is the syntax.</p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> These /vsicurl/option1=val1[,<wbr>optionN=valN]*,url=<a href="http://example.com/foo.tif" target="_blank">http://<wbr>example.com/foo.tif</a></p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> identifiers (or filenames or whatever we call them) may spread from GDAL</p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> into the wider geospatial programming domain. Speaking from my experience</p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> with Rasterio, open source Python GIS developers expect the /vsi* filenames</p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> to "just work" in all software. Can we consider using a more standard</p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> syntax? One that has parsers already deployed everywhere?</p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px"> </p>
</span><p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">I don't really see a use of parsing those /vsi names by user code. User code has to compose them, not parse them. But I can see your point for something more standardized.</p><span class="">
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px"> </p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> That syntax gives the /vsi* filenames the form of a "reflector" URL such as</p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> we see in Google searches (for example:</p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> <a href="https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=1&ved=0ahUKEwj" target="_blank">https://www.google.com/url?sa=<wbr>t&rct=j&q=&esrc=s&source=web&<wbr>cd=1&ved=0ahUKEwj</a></p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> C6e7hvevWAhXmjFQKHWsHDyMQFggmM<wbr>AA&url=http%3A%2F%<a href="http://2Fwww.gdal.org" target="_blank">2Fwww.gdal.<wbr>org</a>%2F&usg=AOvVaw</p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> 3fbRv5TusYwkXgz2Acf2kt) and there are abundant tools and a body of knowledge</p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> about how to parse and work with these.</p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px"> </p>
</span><p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">One downside is that you need to URLEncode the URL, which can make it painful when composing it at hand.</p><span class="">
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px"> </p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px"> </p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">-- </p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">Spatialys - Geospatial professional services</p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px"><a href="http://www.spatialys.com" target="_blank">http://www.spatialys.com</a></p></span></div><br>______________________________<wbr>_________________<br>
gdal-dev mailing list<br>
<a href="mailto:gdal-dev@lists.osgeo.org">gdal-dev@lists.osgeo.org</a><br>
<a href="https://lists.osgeo.org/mailman/listinfo/gdal-dev" rel="noreferrer" target="_blank">https://lists.osgeo.org/<wbr>mailman/listinfo/gdal-dev</a><br></blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div style="color:rgb(0,0,0);font-family:Helvetica;font-size:12px">Regards,</div><div style="color:rgb(0,0,0);font-family:Helvetica;font-size:12px">Craig</div><div style="color:rgb(0,0,0);font-family:Helvetica;font-size:12px"><br></div><div style="color:rgb(0,0,0);font-family:Helvetica;font-size:12px">Developer</div><div style="color:rgb(0,0,0);font-family:Helvetica;font-size:12px">Koordinates</div><div style="color:rgb(0,0,0);font-family:Helvetica;font-size:12px"><br></div><div style="color:rgb(0,0,0);font-family:Helvetica;font-size:12px"><a href="tel:+64%2021%20256%209488" style="color:rgb(17,85,204)" target="_blank">+64 21 256 9488</a> / <a href="http://koordinates.com/" style="color:rgb(17,85,204)" target="_blank">koordinates.com</a> / <a href="https://twitter.com/koordinates" style="color:rgb(17,85,204)" target="_blank">@koordinates</a></div></div></div>
</div></div>