<div dir="ltr"><div dir="ltr">Howard,<div><br></div><div>I have been looking into this further and I would like to better understand how you envision using FileSpec and VSI to move over from Arbiter.</div><div><br></div><div>Taking the LasReader for example and in particular this line - <a href="https://github.com/PDAL/PDAL/blob/master/io/LasReader.cpp#L257">https://github.com/PDAL/PDAL/blob/master/io/LasReader.cpp#L257</a> where arbiter copies the LAS file locally, a direct replacement with VSI will also download the whole file. A Virtual File System (such as that in TileDB VFS) will allow reading the LAS header and then reading the file in chunks of bytes. I think the latter is preferable and am happy to help add it. This is similar to adding an opener to GDAL or LasPy.</div><div><br></div><div>Norman</div></div></div><br><div class="gmail_quote gmail_quote_container"><div dir="ltr" class="gmail_attr">On Thu, Jan 23, 2025 at 1:25 PM Paul Harwood via pdal <<a href="mailto:pdal@lists.osgeo.org">pdal@lists.osgeo.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="auto">+1</div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, 23 Jan 2025, 19:16 Kirk Waters - NOAA Federal via pdal, <<a href="mailto:pdal@lists.osgeo.org" target="_blank">pdal@lists.osgeo.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Howard,</div><div>I'm in favor of your proposal to switch to VSI. Admittedly, I haven't used Arbiter much, but I have used the VSI mechanisms.</div><div><br clear="all"></div><div><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><font face="arial, helvetica, sans-serif">Kirk Waters, PhD </font></div><div><font face="arial, helvetica, sans-serif">NOAA Office for Coastal Management<br></font></div><div><font face="arial, helvetica, sans-serif">Applied Sciences Program </font></div><div><div><font face="arial, helvetica, sans-serif"><a href="http://coast.noaa.gov/digitalcoast" rel="noreferrer" target="_blank">coast.noaa.gov/digitalcoast</a></font></div></div><div><br></div><div><br></div></div></div></div><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Jan 23, 2025 at 1:48 PM Howard Butler via pdal <<a href="mailto:pdal@lists.osgeo.org" rel="noreferrer" target="_blank">pdal@lists.osgeo.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">All,<br>
<br>
Recent tickets and pull requests to PDAL have shown that Arbiter's capabilities, especially for cloud object storage platforms that aren't AWS, are lacking in a number of areas. PDAL has previously used the Arbiter library [1] from Connor Manning to provide its virtual file access, but it wasn't designed to act in this role from the beginning, and it has evolved piecewise to be where it is today. Do we really want to keep investing to add capability there when it already exists in one of our dependencies?<br>
<br>
GDAL's VSI [2] has also evolved to provide a virtual access layer, but it has more features, a wider testing footprint, and covers many more types of remote/alternative resources. Because PDAL already has a hard GDAL dependency, I'm writing to see if the user community would be enthusiastic about PDAL dropping Arbiter and leveraging VSI going forward.<br>
<br>
A few things could help make this switchover less disruptive. First, PDAL 2.9 will introduce something called a "FileSpec" [3] that will allow the 'filename' in a pipeline to be an object with an arbitrary number of nodes under it. These could include typical configuration options. Second, I suspect that many are not fans of VSI's syntax. We would presumably allow users to specify 'filename' using VSI syntax, but also provide the ability for users to use proper protocol prefixes and define all security and other configuration information via the FileSpec. <br>
<br>
What say you?<br>
<br>
Howard<br>
<br>
[1] <a href="https://github.com/connormanning/arbiter" rel="noreferrer noreferrer" target="_blank">https://github.com/connormanning/arbiter</a><br>
[2] <a href="https://gdal.org/en/stable/user/virtual_file_systems.html" rel="noreferrer noreferrer" target="_blank">https://gdal.org/en/stable/user/virtual_file_systems.html</a><br>
[3] <a href="https://github.com/PDAL/PDAL/pull/4625" rel="noreferrer noreferrer" target="_blank">https://github.com/PDAL/PDAL/pull/4625</a><br>
_______________________________________________<br>
pdal mailing list<br>
<a href="mailto:pdal@lists.osgeo.org" rel="noreferrer" target="_blank">pdal@lists.osgeo.org</a><br>
<a href="https://lists.osgeo.org/mailman/listinfo/pdal" rel="noreferrer noreferrer" target="_blank">https://lists.osgeo.org/mailman/listinfo/pdal</a><br>
</blockquote></div>
_______________________________________________<br>
pdal mailing list<br>
<a href="mailto:pdal@lists.osgeo.org" rel="noreferrer" target="_blank">pdal@lists.osgeo.org</a><br>
<a href="https://lists.osgeo.org/mailman/listinfo/pdal" rel="noreferrer noreferrer" target="_blank">https://lists.osgeo.org/mailman/listinfo/pdal</a><br>
</blockquote></div>
_______________________________________________<br>
pdal mailing list<br>
<a href="mailto:pdal@lists.osgeo.org" target="_blank">pdal@lists.osgeo.org</a><br>
<a href="https://lists.osgeo.org/mailman/listinfo/pdal" rel="noreferrer" target="_blank">https://lists.osgeo.org/mailman/listinfo/pdal</a><br>
</blockquote></div>