[gdal-dev] Optional rust library as a dependency of the Zarr driver

[Greg Troxel]

Even Rouault via gdal-dev <gdal-dev at lists.osgeo.org> writes:

> Hi,
>
> We are in the situation where we are about to have our first Rust
> direct (optional) dependency. This is for the "pcodec" lossless codec
> (https://github.com/pcodec/pcodec) that is used by some Zarr
> datasets. It offers a "cpcodec" crate with C bindings which we can
> use.
>
> I've tried in https://github.com/OSGeo/gdal/pull/14779 to offer
> different options:
>
> - by default to satisfy "casual" developers building GDAL who are
>   mostly seeking ease of build and are fine with automatic download of
>   dependencies (pcodec and corrosion-rs for the CMake rust
>   integration), using git at CMake time,

I am very opposed to this as a default, and really I am opposd to this
at all.  Programs simply should not download anything at build time.
This is not allowed in packaging, and it's a lot of why we have had so
many npm supply chain attacks.  I don't think anybody should be ok with
build-time downloads, and I don't think we should enable that.  Yes, I
know there might be checksums, but I think it's still a problem.

If we want to let people do this, they should have to pass some
argument/command to authorize downloading this library.  That's not
hard, and people are going to have to build, because packagers are not
going to turn on download stuff at build time.  Except IDK about
osgeo4w.

> - as well as offering more control with manual settings - either by
>   pointing directly to an already installed cpcodec library, or by
>   pointing to the locations of the source tree of corrosion-rs and
>   pcodec - that should be more compatible of policies/constraints
>  of packagers.
>
> Review of people who might have integrated Rust crates in CMake C/C++
> projects welcome. See the PR description for details.

Now we see the violence inherent in the system.  Rust acts like their
crate world is ok, and it's not, and it torques everybody else around.
Many languages expect everyone to accomodate their special
almost-a-packaging-system silo that only works in their language, and to
treat them a special.  That puts you in a bad place, I realize.

The philosophically right approach would be to have a C-interface*
library that uses the rust-world normal build stuff (which packagers
don't like but have to adapt to and have) and have that be packaged.
Then it's just something to look for with cmake; it logically should not
be gdal's problem that it is written in rust any more than gdal should
have to know that e.g. sqlite[iff is in C.

I think a good example is librsvg.  It's a library, written significantl
in rust, but programs that use it just look for a .h and link against
it.  They aren't forced to act like it's all special because it used
rust.

Is pcodec packaged in Debian or any place else?  I can have a look at
adding it to pkgsrc, and see how that goes.  We certainly have a lot of
things in rust.

We also are set up for programs that are fully in rust to fetch crates
at packaging time and add them to the manifest of source tarballs.  But
I am very wary of trying to go down that path for a mostly-C++ program.

Deoes gdal have a prior history of dealing with dependencies that should
be packaged but aren't?

I wonder if the plugin system is the answer here, to say that anything
with troublesome (not packaged and checkable with cmake without special
help) dependencies should be a plugin maintained in a separate repo.
But I am not sure if the plugin system is fully workable, in terms of if
flipping it on and trying to do everything that way really works, just
because I haven't tried.

* I see C interface as really native ABI on the platform, more than it
  is C, but I realize that's an arguable point of view.