[gdal-dev] Un-vendoring a number of third-party libraries?

[Greg Troxel]

Even Rouault via gdal-dev <gdal-dev at lists.osgeo.org> writes:

> I'm considering removing from the source tree a number of third-party
> libraries that we have vendored over the years: zlib, libpng, libjpeg,
> giflib, liblerc

I'm basically strongly in favor of un-vendoring.  I view vendoring as a
bug, even if it is a workaround for other bugs.

> I believe the main reason for having them vendored is now mostly
> historical, dating back to times where there was no packaging system
> on Windows. Now we have Conda-Forge or vcpkg, it is easy to have those
> dependencies installed.

>From the pkgrsc viewpoint, I don't see any reasons why vendoring helps.
It looks to me like the build is using normal dependencies, including
zlib, jpeg, tiff, geotiff.  But apparently not shapelib and json-c.

$ ldd /usr/pkg/lib/libgdal.so
/usr/pkg/lib/libgdal.so:
        -lgeos_c.1 => /usr/pkg/lib/libgeos_c.so.1
        -lgeos.3.12.1 => /usr/pkg/lib/libgeos.so.3.12.1
        -lstdc++.9 => /usr/lib/libstdc++.so.9
        -lm.0 => /usr/lib/libm.so.0
        -lc.12 => /usr/lib/libc.so.12
        -lgcc_s.1 => /usr/lib/libgcc_s.so.1
        -lwebp.7 => /usr/pkg/lib/libwebp.so.7
        -lsharpyuv.0 => /usr/pkg/lib/libsharpyuv.so.0
        -lpthread.1 => /usr/lib/libpthread.so.1
        -lexpat.2 => /usr/lib/libexpat.so.2
        -lxerces-c-3.2 => /usr/pkg/lib/libxerces-c-3.2.so
        -lopenjp2.7 => /usr/pkg/lib/libopenjp2.so.7
        -lnetcdf.19 => /usr/pkg/lib/libnetcdf.so.19
        -lexecinfo.0 => /usr/lib/libexecinfo.so.0
        -lelf.2 => /usr/lib/libelf.so.2
        -lhdf5_hl.200 => /usr/pkg/lib/libhdf5_hl.so.200
        -lhdf5.200 => /usr/pkg/lib/libhdf5.so.200
        -lsz.2 => /usr/pkg/lib/libsz.so.2
        -laec.0 => /usr/pkg/lib/libaec.so.0
        -lz.1 => /usr/lib/libz.so.1
        -lbz2.1 => /usr/lib/libbz2.so.1
        -lxml2.2 => /usr/pkg/lib/libxml2.so.2
        -llzma.2 => /usr/lib/liblzma.so.2
        -lcurl.4 => /usr/pkg/lib/libcurl.so.4
        -lnghttp2.14 => /usr/pkg/lib/libnghttp2.so.14
        -lidn2.0 => /usr/pkg/lib/libidn2.so.0
        -lunistring.5 => /usr/pkg/lib/libunistring.so.5
        -lintl.1 => /usr/lib/libintl.so.1
        -lssl.15 => /usr/lib/libssl.so.15
        -lcrypto.15 => /usr/lib/libcrypto.so.15
        -lcrypt.1 => /lib/libcrypt.so.1
        -lgssapi.12 => /usr/lib/libgssapi.so.12
        -lkrb5.28 => /usr/lib/libkrb5.so.28
        -lhx509.7 => /usr/lib/libhx509.so.7
        -lasn1.10 => /usr/lib/libasn1.so.10
        -lcom_err.8 => /usr/lib/libcom_err.so.8
        -lroken.20 => /usr/lib/libroken.so.20
        -lutil.7 => /usr/lib/libutil.so.7
        -lwind.1 => /usr/lib/libwind.so.1
        -lheimbase.2 => /usr/lib/libheimbase.so.2
        -lheimntlm.6 => /usr/lib/libheimntlm.so.6
        -lgif.7 => /usr/pkg/lib/libgif.so.7
        -lgeotiff.5 => /usr/pkg/lib/libgeotiff.so.5
        -lproj.19 => /usr/pkg/lib/libproj.so.19
        -lsqlite3.1 => /usr/lib/libsqlite3.so.1
        -ltiff.6 => /usr/pkg/lib/libtiff.so.6
        -ljbig.2 => /usr/pkg/lib/libjbig.so.2
        -ljpeg.8 => /usr/pkg/lib/libjpeg.so.8
        -lpng16.16 => /usr/pkg/lib/libpng16.so.16
        -lrt.1 => /usr/lib/librt.so.1
        -lpcre.1 => /usr/pkg/lib/libpcre.so.1
        -lqhull_r.8.0 => /usr/pkg/lib/libqhull_r.so.8.0

> For libjpeg, there was a particular history related to 12-bit JPEG

So documentation of prereqs will say "you need jpeg, and you really
should use jpeg-turbo so you have 12-bit support"?  If so, sounds good.

> Benefits of un-vendoring those libraries:
>
> - currently, we must take care of updating them regularly, in
>   particular to make sure they integrate the latest fixes for their
>  vulnerabilities.

And not just that, but to have a gdal point release within days of any
vendored upstream release that has a security fix, documented as such or
not.  That's a tall order, and ~nobody achieves it.  This point is
strongly in favor of unvendoring.

Also, if there is a portablity or other bug fix, that needs to be
imported and apoint release, so people can get those fixes.

> Looking a bit around in different open source build recipees of GDAL
> (Debian, Conda-Forge, vcpkg, OSGeo4W, gisinternals, rasterio-wheels),
> those proposed changes should have modest impact, as they already
> mostly use external libraries. What I've identified (I may have missed
> things) to require changes from the maintainers of those distributions
> to keep the same level of functionality:
>
> - gisinternals doesn't seem to have a liblerc build
>
> - rasterio-wheels doesn't seem to have libpng and giflib builds

I am pretty sure pkgsrc will have little to no trouble.  There's no
liblerc, but it's optional and surely it can be packaged if somebody
cares.

> Potential candidates, but would remain in-tree for now:
>
> - libtiff: compulsory dependency. GDAL has been the main driver for
>   most libtiff development over the last 10 years, and GDAL autotest
>   suite tortures libtiff much more than libtiff own testsuite, hence
>   it is quite convenient to have the capability of vendoring it.  Plus
>   the fact that for "staging codecs" (that is codecs not yet
>   integrated in official libtiff), currently JPEG-XL (a few years ago
>   this was the LERC codec), we can't build them against an external
>  libtiff.

I see this as a workaround and it would be better if it weren't needed,
but getting most of the gain and leaving the harder issues for later is
a great strategy.  FWIW pkgsrc's gdal build uses pkgsrc libtiff and I
read normal tiffs just fine.

> - shapelib. compulsory dependency.  External default shapelib build
>   uses 32-bit file offset, whereas the internal shapelib is built with
>   64-bit offset support (to use .DBF files > 2 GB). We don't have
>   build support for using it as external lib.

Wow, that seems like quite a mess.  I wonder if that's only for
operating systems that have a LFS notion, vs those that just have 64-bit
off_t?


Overall, I would say unvendor as much as you feel you can do without
real trouble, and then repeat, and from my viewpoint, the more and
sooner the better, but I realize others have issues.