[pdal] Replacing Arbiter with GDAL's VSI?

[Norman Barker]

Howard,

I am in favour of moving to GDAL VSI as it has been well tested. There are
scenarios around chaining credentials that are not supported but most of
the functionality is there.

I am in favour of an `opener` style such as that in laspy and rasterio and
I wonder if that could be a file stage in PDAL. For now I use the Python
bindings and pass in numpy arrays to the pipeline.

An opener stage would also enable the use of TileDB VFS -
https://docs.tiledb.com/main/how-to/virtual-filesystem which allows access
to all files on remote storage (not just tiledb arrays) and could also be
used as a sink to live streams.

Norman

On Thu, Jan 23, 2025 at 12:56 PM Howard Butler via pdal <
pdal at lists.osgeo.org> wrote:

> All,
>
> Recent tickets and pull requests to PDAL have shown that Arbiter's
> capabilities, especially for cloud object storage platforms that aren't
> AWS, are lacking in a number of areas. PDAL has previously used the Arbiter
> library [1] from Connor Manning to provide its virtual file access, but it
> wasn't designed to act in this role from the beginning, and it has evolved
> piecewise to be where it is today. Do we really want to keep investing to
> add capability there when it already exists in one of our dependencies?
>
> GDAL's VSI [2] has also evolved to provide a virtual access layer, but it
> has more features, a wider testing footprint, and covers many more types of
> remote/alternative resources. Because PDAL already has a hard GDAL
> dependency, I'm writing to see if the user community would be enthusiastic
> about PDAL dropping Arbiter and leveraging VSI going forward.
>
> A few things could help make this switchover less disruptive. First, PDAL
> 2.9 will introduce something called a "FileSpec" [3] that will allow the
> 'filename' in a pipeline to be an object with an arbitrary number of nodes
> under it. These could include typical configuration options. Second, I
> suspect that many are not fans of VSI's syntax. We would presumably allow
> users to specify 'filename' using VSI syntax, but also provide the ability
> for users to use proper protocol prefixes and define all security and other
> configuration information via the FileSpec.
>
> What say you?
>
> Howard
>
> [1] https://github.com/connormanning/arbiter
> [2] https://gdal.org/en/stable/user/virtual_file_systems.html
> [3] https://github.com/PDAL/PDAL/pull/4625
> _______________________________________________
> pdal mailing list
> pdal at lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/pdal
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/pdal/attachments/20250123/782005d1/attachment-0001.htm>