[pdal] Replacing Arbiter with GDAL's VSI?

[Norman Barker]

Howard,

I have been looking into this further and I would like to better understand
how you envision using FileSpec and VSI to move over from Arbiter.

Taking the LasReader for example and in particular this line -
https://github.com/PDAL/PDAL/blob/master/io/LasReader.cpp#L257 where
arbiter copies the LAS file locally, a direct replacement with VSI will
also download the whole file. A Virtual File System (such as that in TileDB
VFS) will allow reading the LAS header and then reading the file in chunks
of bytes. I think the latter is preferable and am happy to help add it.
This is similar to adding an opener to GDAL or LasPy.

Norman

On Thu, Jan 23, 2025 at 1:25 PM Paul Harwood via pdal <pdal at lists.osgeo.org>
wrote:

> +1
>
> On Thu, 23 Jan 2025, 19:16 Kirk Waters - NOAA Federal via pdal, <
> pdal at lists.osgeo.org> wrote:
>
>> Howard,
>> I'm in favor of your proposal to switch to VSI. Admittedly, I haven't
>> used Arbiter much, but I have used the VSI mechanisms.
>>
>> Kirk Waters, PhD
>> NOAA Office for Coastal Management
>> Applied Sciences Program
>> coast.noaa.gov/digitalcoast
>>
>>
>>
>>
>> On Thu, Jan 23, 2025 at 1:48 PM Howard Butler via pdal <
>> pdal at lists.osgeo.org> wrote:
>>
>>> All,
>>>
>>> Recent tickets and pull requests to PDAL have shown that Arbiter's
>>> capabilities, especially for cloud object storage platforms that aren't
>>> AWS, are lacking in a number of areas. PDAL has previously used the Arbiter
>>> library [1] from Connor Manning to provide its virtual file access, but it
>>> wasn't designed to act in this role from the beginning, and it has evolved
>>> piecewise to be where it is today. Do we really want to keep investing to
>>> add capability there when it already exists in one of our dependencies?
>>>
>>> GDAL's VSI [2] has also evolved to provide a virtual access layer, but
>>> it has more features, a wider testing footprint, and covers many more types
>>> of remote/alternative resources. Because PDAL already has a hard GDAL
>>> dependency, I'm writing to see if the user community would be enthusiastic
>>> about PDAL dropping Arbiter and leveraging VSI going forward.
>>>
>>> A few things could help make this switchover less disruptive. First,
>>> PDAL 2.9 will introduce something called a "FileSpec" [3] that will allow
>>> the 'filename' in a pipeline to be an object with an arbitrary number of
>>> nodes under it. These could include typical configuration options. Second,
>>> I suspect that many are not fans of VSI's syntax. We would presumably allow
>>> users to specify 'filename' using VSI syntax, but also provide the ability
>>> for users to use proper protocol prefixes and define all security and other
>>> configuration information via the FileSpec.
>>>
>>> What say you?
>>>
>>> Howard
>>>
>>> [1] https://github.com/connormanning/arbiter
>>> [2] https://gdal.org/en/stable/user/virtual_file_systems.html
>>> [3] https://github.com/PDAL/PDAL/pull/4625
>>> _______________________________________________
>>> pdal mailing list
>>> pdal at lists.osgeo.org
>>> https://lists.osgeo.org/mailman/listinfo/pdal
>>>
>> _______________________________________________
>> pdal mailing list
>> pdal at lists.osgeo.org
>> https://lists.osgeo.org/mailman/listinfo/pdal
>>
> _______________________________________________
> pdal mailing list
> pdal at lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/pdal
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/pdal/attachments/20250129/0613ade6/attachment.htm>