[pdal] Replacing Arbiter with GDAL's VSI?

[Howard Butler]

All,

Recent tickets and pull requests to PDAL have shown that Arbiter's capabilities, especially for cloud object storage platforms that aren't AWS, are lacking in a number of areas. PDAL has previously used the Arbiter library [1] from Connor Manning to provide its virtual file access, but it wasn't designed to act in this role from the beginning, and it has evolved piecewise to be where it is today. Do we really want to keep investing to add capability there when it already exists in one of our dependencies?

GDAL's VSI [2] has also evolved to provide a virtual access layer, but it has more features, a wider testing footprint, and covers many more types of remote/alternative resources. Because PDAL already has a hard GDAL dependency, I'm writing to see if the user community would be enthusiastic about PDAL dropping Arbiter and leveraging VSI going forward.

A few things could help make this switchover less disruptive. First, PDAL 2.9 will introduce something called a "FileSpec" [3] that will allow the 'filename' in a pipeline to be an object with an arbitrary number of nodes under it. These could include typical configuration options. Second, I suspect that many are not fans of VSI's syntax. We would presumably allow users to specify 'filename' using VSI syntax, but also provide the ability for users to use proper protocol prefixes and define all security and other configuration information via the FileSpec. 

What say you?

Howard

[1] https://github.com/connormanning/arbiter
[2] https://gdal.org/en/stable/user/virtual_file_systems.html
[3] https://github.com/PDAL/PDAL/pull/4625