[gdal-dev] Un-vendoring a number of third-party libraries?
Kurt Schwehr
schwehr at gmail.com
Fri Dec 15 22:40:30 PST 2023
+1 for un-vendoring. Long term, I think that will be a big win. Looks like
others have covered all of the issues that I can think of.
On Fri, Dec 15, 2023 at 4:45 PM Greg Troxel via gdal-dev <
gdal-dev at lists.osgeo.org> wrote:
> Even Rouault via gdal-dev <gdal-dev at lists.osgeo.org> writes:
>
> > I'm considering removing from the source tree a number of third-party
> > libraries that we have vendored over the years: zlib, libpng, libjpeg,
> > giflib, liblerc
>
> I'm basically strongly in favor of un-vendoring. I view vendoring as a
> bug, even if it is a workaround for other bugs.
>
> > I believe the main reason for having them vendored is now mostly
> > historical, dating back to times where there was no packaging system
> > on Windows. Now we have Conda-Forge or vcpkg, it is easy to have those
> > dependencies installed.
>
> From the pkgrsc viewpoint, I don't see any reasons why vendoring helps.
> It looks to me like the build is using normal dependencies, including
> zlib, jpeg, tiff, geotiff. But apparently not shapelib and json-c.
>
> $ ldd /usr/pkg/lib/libgdal.so
> /usr/pkg/lib/libgdal.so:
> -lgeos_c.1 => /usr/pkg/lib/libgeos_c.so.1
> -lgeos.3.12.1 => /usr/pkg/lib/libgeos.so.3.12.1
> -lstdc++.9 => /usr/lib/libstdc++.so.9
> -lm.0 => /usr/lib/libm.so.0
> -lc.12 => /usr/lib/libc.so.12
> -lgcc_s.1 => /usr/lib/libgcc_s.so.1
> -lwebp.7 => /usr/pkg/lib/libwebp.so.7
> -lsharpyuv.0 => /usr/pkg/lib/libsharpyuv.so.0
> -lpthread.1 => /usr/lib/libpthread.so.1
> -lexpat.2 => /usr/lib/libexpat.so.2
> -lxerces-c-3.2 => /usr/pkg/lib/libxerces-c-3.2.so
> -lopenjp2.7 => /usr/pkg/lib/libopenjp2.so.7
> -lnetcdf.19 => /usr/pkg/lib/libnetcdf.so.19
> -lexecinfo.0 => /usr/lib/libexecinfo.so.0
> -lelf.2 => /usr/lib/libelf.so.2
> -lhdf5_hl.200 => /usr/pkg/lib/libhdf5_hl.so.200
> -lhdf5.200 => /usr/pkg/lib/libhdf5.so.200
> -lsz.2 => /usr/pkg/lib/libsz.so.2
> -laec.0 => /usr/pkg/lib/libaec.so.0
> -lz.1 => /usr/lib/libz.so.1
> -lbz2.1 => /usr/lib/libbz2.so.1
> -lxml2.2 => /usr/pkg/lib/libxml2.so.2
> -llzma.2 => /usr/lib/liblzma.so.2
> -lcurl.4 => /usr/pkg/lib/libcurl.so.4
> -lnghttp2.14 => /usr/pkg/lib/libnghttp2.so.14
> -lidn2.0 => /usr/pkg/lib/libidn2.so.0
> -lunistring.5 => /usr/pkg/lib/libunistring.so.5
> -lintl.1 => /usr/lib/libintl.so.1
> -lssl.15 => /usr/lib/libssl.so.15
> -lcrypto.15 => /usr/lib/libcrypto.so.15
> -lcrypt.1 => /lib/libcrypt.so.1
> -lgssapi.12 => /usr/lib/libgssapi.so.12
> -lkrb5.28 => /usr/lib/libkrb5.so.28
> -lhx509.7 => /usr/lib/libhx509.so.7
> -lasn1.10 => /usr/lib/libasn1.so.10
> -lcom_err.8 => /usr/lib/libcom_err.so.8
> -lroken.20 => /usr/lib/libroken.so.20
> -lutil.7 => /usr/lib/libutil.so.7
> -lwind.1 => /usr/lib/libwind.so.1
> -lheimbase.2 => /usr/lib/libheimbase.so.2
> -lheimntlm.6 => /usr/lib/libheimntlm.so.6
> -lgif.7 => /usr/pkg/lib/libgif.so.7
> -lgeotiff.5 => /usr/pkg/lib/libgeotiff.so.5
> -lproj.19 => /usr/pkg/lib/libproj.so.19
> -lsqlite3.1 => /usr/lib/libsqlite3.so.1
> -ltiff.6 => /usr/pkg/lib/libtiff.so.6
> -ljbig.2 => /usr/pkg/lib/libjbig.so.2
> -ljpeg.8 => /usr/pkg/lib/libjpeg.so.8
> -lpng16.16 => /usr/pkg/lib/libpng16.so.16
> -lrt.1 => /usr/lib/librt.so.1
> -lpcre.1 => /usr/pkg/lib/libpcre.so.1
> -lqhull_r.8.0 => /usr/pkg/lib/libqhull_r.so.8.0
>
> > For libjpeg, there was a particular history related to 12-bit JPEG
>
> So documentation of prereqs will say "you need jpeg, and you really
> should use jpeg-turbo so you have 12-bit support"? If so, sounds good.
>
> > Benefits of un-vendoring those libraries:
> >
> > - currently, we must take care of updating them regularly, in
> > particular to make sure they integrate the latest fixes for their
> > vulnerabilities.
>
> And not just that, but to have a gdal point release within days of any
> vendored upstream release that has a security fix, documented as such or
> not. That's a tall order, and ~nobody achieves it. This point is
> strongly in favor of unvendoring.
>
> Also, if there is a portablity or other bug fix, that needs to be
> imported and apoint release, so people can get those fixes.
>
> > Looking a bit around in different open source build recipees of GDAL
> > (Debian, Conda-Forge, vcpkg, OSGeo4W, gisinternals, rasterio-wheels),
> > those proposed changes should have modest impact, as they already
> > mostly use external libraries. What I've identified (I may have missed
> > things) to require changes from the maintainers of those distributions
> > to keep the same level of functionality:
> >
> > - gisinternals doesn't seem to have a liblerc build
> >
> > - rasterio-wheels doesn't seem to have libpng and giflib builds
>
> I am pretty sure pkgsrc will have little to no trouble. There's no
> liblerc, but it's optional and surely it can be packaged if somebody
> cares.
>
> > Potential candidates, but would remain in-tree for now:
> >
> > - libtiff: compulsory dependency. GDAL has been the main driver for
> > most libtiff development over the last 10 years, and GDAL autotest
> > suite tortures libtiff much more than libtiff own testsuite, hence
> > it is quite convenient to have the capability of vendoring it. Plus
> > the fact that for "staging codecs" (that is codecs not yet
> > integrated in official libtiff), currently JPEG-XL (a few years ago
> > this was the LERC codec), we can't build them against an external
> > libtiff.
>
> I see this as a workaround and it would be better if it weren't needed,
> but getting most of the gain and leaving the harder issues for later is
> a great strategy. FWIW pkgsrc's gdal build uses pkgsrc libtiff and I
> read normal tiffs just fine.
>
> > - shapelib. compulsory dependency. External default shapelib build
> > uses 32-bit file offset, whereas the internal shapelib is built with
> > 64-bit offset support (to use .DBF files > 2 GB). We don't have
> > build support for using it as external lib.
>
> Wow, that seems like quite a mess. I wonder if that's only for
> operating systems that have a LFS notion, vs those that just have 64-bit
> off_t?
>
>
> Overall, I would say unvendor as much as you feel you can do without
> real trouble, and then repeat, and from my viewpoint, the more and
> sooner the better, but I realize others have issues.
> _______________________________________________
> gdal-dev mailing list
> gdal-dev at lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/gdal-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20231215/40db4456/attachment.htm>
More information about the gdal-dev
mailing list