[pdal] Proposal: Remove SQLite reader/writer for PDAL 2.0

[Jed Frechette]

On Wed, 31 Jul 2019 08:28:09 -0500 Howard Butler wrote:
> On second thought, we'll table the removal until 2.1 to allow a full
> release cycle for everyone who might be impacted to catch up, but we will
> still plan to remove these drivers. Again, if you are going to be impacted
> by this change we would like to hear from you. We don't know of much use of
> these drivers, and their continued maintenance burden does not seem worth
> it given their limited use.

Good timing! In the next few days we were about to start building some
tools based on them.

The use case was to round-trip data to PDAL and back from inside a
couple of applications that include embedded Python interpreters, but
have limited support for other PDAL formats. In particular, although
the specific applications we're working with do have some level of
support for both las and e57, the built in readers and writers don't
always handle arbitrary dimensions well, if at all.

Since an SQLite interface is included in the Python Standard Library
the idea was to use that rather than trying to muck about with adding
external modules to each application's Python install (and the
maintenance nightmare that entails). Based on PDAL's current docs,
SQLite looked like a good candidate for storing points with arbitrary
dimensions in a way that was both well supported by PDAL and easily
accessible by anything with a Python interpreter. If SQLite is
deprecated, then good old text files look like about the only other
container that would fill that niche.

I should note that all of this was theoretical and we may well have
run in to issues trying to put it in to practice. Nonetheless, it
seemed like a reasonable approach and knowing that at least one
commercial lidar software had used their own flavor of SQLite as a
native file format gave me some additional confidence that it wan't a
bad solution.

As an aside, I don't know if it makes sense for PDAL to have a
"native" container format or not, but it would be helpful to more
clearly document which writer/reader pairs can be expected to
losslessly round-trip data if PDAL is the only application involved.
With the number of useful filters PDAL already has, I don't think it
is unreasonable to start thinking about it as a central part of a
processing workflow rather than just  a tool for moving data from
External Application A to External Application B. The more it takes on
a central role the more I think it makes sense to be asking "What's
the best container format to use with PDAL?" rather than just
conforming to whatever might be supported by external applications.

Best wishes,

-- 
Jed Frechette