<div dir="ltr">+1 for un-vendoring. Long term, I think that will be a big win. Looks like others have covered all of the issues that I can think of.<br><div><br></div><div><br></div><div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Dec 15, 2023 at 4:45 PM Greg Troxel via gdal-dev <<a href="mailto:gdal-dev@lists.osgeo.org">gdal-dev@lists.osgeo.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Even Rouault via gdal-dev <<a href="mailto:gdal-dev@lists.osgeo.org" target="_blank">gdal-dev@lists.osgeo.org</a>> writes:<br>
<br>
> I'm considering removing from the source tree a number of third-party<br>
> libraries that we have vendored over the years: zlib, libpng, libjpeg,<br>
> giflib, liblerc<br>
<br>
I'm basically strongly in favor of un-vendoring. I view vendoring as a<br>
bug, even if it is a workaround for other bugs.<br>
<br>
> I believe the main reason for having them vendored is now mostly<br>
> historical, dating back to times where there was no packaging system<br>
> on Windows. Now we have Conda-Forge or vcpkg, it is easy to have those<br>
> dependencies installed.<br>
<br>
>From the pkgrsc viewpoint, I don't see any reasons why vendoring helps.<br>
It looks to me like the build is using normal dependencies, including<br>
zlib, jpeg, tiff, geotiff. But apparently not shapelib and json-c.<br>
<br>
$ ldd /usr/pkg/lib/libgdal.so<br>
/usr/pkg/lib/libgdal.so:<br>
-lgeos_c.1 => /usr/pkg/lib/libgeos_c.so.1<br>
-lgeos.3.12.1 => /usr/pkg/lib/libgeos.so.3.12.1<br>
-lstdc++.9 => /usr/lib/libstdc++.so.9<br>
-lm.0 => /usr/lib/libm.so.0<br>
-lc.12 => /usr/lib/libc.so.12<br>
-lgcc_s.1 => /usr/lib/libgcc_s.so.1<br>
-lwebp.7 => /usr/pkg/lib/libwebp.so.7<br>
-lsharpyuv.0 => /usr/pkg/lib/libsharpyuv.so.0<br>
-lpthread.1 => /usr/lib/libpthread.so.1<br>
-lexpat.2 => /usr/lib/libexpat.so.2<br>
-lxerces-c-3.2 => /usr/pkg/lib/<a href="http://libxerces-c-3.2.so" rel="noreferrer" target="_blank">libxerces-c-3.2.so</a><br>
-lopenjp2.7 => /usr/pkg/lib/libopenjp2.so.7<br>
-lnetcdf.19 => /usr/pkg/lib/libnetcdf.so.19<br>
-lexecinfo.0 => /usr/lib/libexecinfo.so.0<br>
-lelf.2 => /usr/lib/libelf.so.2<br>
-lhdf5_hl.200 => /usr/pkg/lib/libhdf5_hl.so.200<br>
-lhdf5.200 => /usr/pkg/lib/libhdf5.so.200<br>
-lsz.2 => /usr/pkg/lib/libsz.so.2<br>
-laec.0 => /usr/pkg/lib/libaec.so.0<br>
-lz.1 => /usr/lib/libz.so.1<br>
-lbz2.1 => /usr/lib/libbz2.so.1<br>
-lxml2.2 => /usr/pkg/lib/libxml2.so.2<br>
-llzma.2 => /usr/lib/liblzma.so.2<br>
-lcurl.4 => /usr/pkg/lib/libcurl.so.4<br>
-lnghttp2.14 => /usr/pkg/lib/libnghttp2.so.14<br>
-lidn2.0 => /usr/pkg/lib/libidn2.so.0<br>
-lunistring.5 => /usr/pkg/lib/libunistring.so.5<br>
-lintl.1 => /usr/lib/libintl.so.1<br>
-lssl.15 => /usr/lib/libssl.so.15<br>
-lcrypto.15 => /usr/lib/libcrypto.so.15<br>
-lcrypt.1 => /lib/libcrypt.so.1<br>
-lgssapi.12 => /usr/lib/libgssapi.so.12<br>
-lkrb5.28 => /usr/lib/libkrb5.so.28<br>
-lhx509.7 => /usr/lib/libhx509.so.7<br>
-lasn1.10 => /usr/lib/libasn1.so.10<br>
-lcom_err.8 => /usr/lib/libcom_err.so.8<br>
-lroken.20 => /usr/lib/libroken.so.20<br>
-lutil.7 => /usr/lib/libutil.so.7<br>
-lwind.1 => /usr/lib/libwind.so.1<br>
-lheimbase.2 => /usr/lib/libheimbase.so.2<br>
-lheimntlm.6 => /usr/lib/libheimntlm.so.6<br>
-lgif.7 => /usr/pkg/lib/libgif.so.7<br>
-lgeotiff.5 => /usr/pkg/lib/libgeotiff.so.5<br>
-lproj.19 => /usr/pkg/lib/libproj.so.19<br>
-lsqlite3.1 => /usr/lib/libsqlite3.so.1<br>
-ltiff.6 => /usr/pkg/lib/libtiff.so.6<br>
-ljbig.2 => /usr/pkg/lib/libjbig.so.2<br>
-ljpeg.8 => /usr/pkg/lib/libjpeg.so.8<br>
-lpng16.16 => /usr/pkg/lib/libpng16.so.16<br>
-lrt.1 => /usr/lib/librt.so.1<br>
-lpcre.1 => /usr/pkg/lib/libpcre.so.1<br>
-lqhull_r.8.0 => /usr/pkg/lib/libqhull_r.so.8.0<br>
<br>
> For libjpeg, there was a particular history related to 12-bit JPEG<br>
<br>
So documentation of prereqs will say "you need jpeg, and you really<br>
should use jpeg-turbo so you have 12-bit support"? If so, sounds good.<br>
<br>
> Benefits of un-vendoring those libraries:<br>
><br>
> - currently, we must take care of updating them regularly, in<br>
> particular to make sure they integrate the latest fixes for their<br>
> vulnerabilities.<br>
<br>
And not just that, but to have a gdal point release within days of any<br>
vendored upstream release that has a security fix, documented as such or<br>
not. That's a tall order, and ~nobody achieves it. This point is<br>
strongly in favor of unvendoring.<br>
<br>
Also, if there is a portablity or other bug fix, that needs to be<br>
imported and apoint release, so people can get those fixes.<br>
<br>
> Looking a bit around in different open source build recipees of GDAL<br>
> (Debian, Conda-Forge, vcpkg, OSGeo4W, gisinternals, rasterio-wheels),<br>
> those proposed changes should have modest impact, as they already<br>
> mostly use external libraries. What I've identified (I may have missed<br>
> things) to require changes from the maintainers of those distributions<br>
> to keep the same level of functionality:<br>
><br>
> - gisinternals doesn't seem to have a liblerc build<br>
><br>
> - rasterio-wheels doesn't seem to have libpng and giflib builds<br>
<br>
I am pretty sure pkgsrc will have little to no trouble. There's no<br>
liblerc, but it's optional and surely it can be packaged if somebody<br>
cares.<br>
<br>
> Potential candidates, but would remain in-tree for now:<br>
><br>
> - libtiff: compulsory dependency. GDAL has been the main driver for<br>
> most libtiff development over the last 10 years, and GDAL autotest<br>
> suite tortures libtiff much more than libtiff own testsuite, hence<br>
> it is quite convenient to have the capability of vendoring it. Plus<br>
> the fact that for "staging codecs" (that is codecs not yet<br>
> integrated in official libtiff), currently JPEG-XL (a few years ago<br>
> this was the LERC codec), we can't build them against an external<br>
> libtiff.<br>
<br>
I see this as a workaround and it would be better if it weren't needed,<br>
but getting most of the gain and leaving the harder issues for later is<br>
a great strategy. FWIW pkgsrc's gdal build uses pkgsrc libtiff and I<br>
read normal tiffs just fine.<br>
<br>
> - shapelib. compulsory dependency. External default shapelib build<br>
> uses 32-bit file offset, whereas the internal shapelib is built with<br>
> 64-bit offset support (to use .DBF files > 2 GB). We don't have<br>
> build support for using it as external lib.<br>
<br>
Wow, that seems like quite a mess. I wonder if that's only for<br>
operating systems that have a LFS notion, vs those that just have 64-bit<br>
off_t?<br>
<br>
<br>
Overall, I would say unvendor as much as you feel you can do without<br>
real trouble, and then repeat, and from my viewpoint, the more and<br>
sooner the better, but I realize others have issues.<br>
_______________________________________________<br>
gdal-dev mailing list<br>
<a href="mailto:gdal-dev@lists.osgeo.org" target="_blank">gdal-dev@lists.osgeo.org</a><br>
<a href="https://lists.osgeo.org/mailman/listinfo/gdal-dev" rel="noreferrer" target="_blank">https://lists.osgeo.org/mailman/listinfo/gdal-dev</a><br>
</blockquote></div>