[gdal-dev] Improving GDAL production in our release

[Kurt Schwehr]

Packaging is hard. There are all sorts of sharp edges. I don't work in the
windows world much at all, so I can't comment on the packaging there, but
there are many package mangers that provide GDAL. And many of us who do
packaging share ideas and fixes across packaging. I personally do bazel
builds that are mostly focused on linux.

(please don't read anything into the order here :)

https://conan.io/center/recipes/gdal
https://anaconda.org/conda-forge/gdal
https://github.com/fink/fink-distributions/blob/master/10.9-libcxx/stable/main/finkinfo/libs/gdal3.info
for MacOSX
https://ports.macports.org/port/gdal/
https://formulae.brew.sh/formula/gdal
https://rpmfind.net/linux/rpm2html/search.php?query=gdal
https://wiki.debian.org/DebianGis
...

Or things that bring in GDAL:

https://rasterio.readthedocs.io/en/stable/installation.html has binary
wheels
https://www.qgis.org/
(and probably some other GIS packages bring along GDAL)
...

I've also collaborated on bash scripts to do custom builds of GDAL. It's a
diverse ecosystem.

-kurt

On Mon, Jan 6, 2025 at 3:52 PM Greg Troxel via gdal-dev <
gdal-dev at lists.osgeo.org> wrote:

> David Klaus <dklaus at carlsonsw.com> writes:
>
> > Thank you for your fast response.
> >
> > Q: You didn't explain whether you are doing import/merge and carrying
> diffs
> > to the sources, and if so, you didn't give a link where others can look
> at
> > them.
> >
> > A: I apologize if I misunderstand your questions. I think you are asking
> > whether or not anyone at our company has modified any of the source GDAL
> > source files or if we are maintaining our own distribution of the source
> > code. If this is the question, the answer is no.
>
> That makes things easier.
>
> > Q: You didn't publish a link to the scripts you are using.
> >
> > A: Personally, I have no problem with sharing the scripts we currently
> use.
> > I will speak with my boss on whether it would be acceptable to share
> those
> > here.
>
> Having been at a large company, I understand.  But nobody here will be
> able to give you advice without reading your script.  And, if you want
> help making your build process better under nondisclosure, you'll need a
> consultant rather than help from the list.
>
> > Q: You didn't explain the problems you are having, just that it's hard.
> >
> > A: My apologies. The problem is that modifying our current scripts in
> order
> > to produce a new build of GDAL takes a significant amount of development
> > time. For example, I am currently working on building Proj-9.5.1 as part
> of
> > our new GDAL build and I'm running into trouble linking "tiff" libraries.
> > This is not likely irreconcilable but it takes time. I see how this can
> be
> > difficult to understand without looking at our current build process. I
> > will discuss this with my boss.
>
> My quick guess is that you are reinventing a packaging system and that
> the last 10% is hard.  That's a little glib, but it sounds like various
> paths.  You might consider using a packaging system, and using a
> different prefix than the regulsr /usr.
>
> You may also want to be very rigorous and organized about -I and -L/-R
> (-R may be spelled something like -Wl,-rpath).
>
> > Q: I hope you are contributing funds to GDAL!  I know that can be hard
> in a
> > corporate environment....
> >
> > A: Of course I have no problem contributing my corporation's money to
> GDAL.
> > Unfortunately, that's why I don't get to make those decisions. However, I
> > will discuss this with my boss. Hopefully, I can persuade him,
>
> You might consider how much the company would pay annually for a
> proprietary license for code of equivalent breadth and quality to gdal,
> proj, etc.  You might look at Oracle's fees, or ESRI's.  And then ask
> management to take 10% of that and donate annually.  What I have always
> found is that companies are ok paying huge amounts of money for
> proprietary licenses, but even with free software advocates inside, it's
> amazingly difficult to donate.
>
> > P.S. You mention the following:
> >
> > "I, and probably others, have scripts that basically set up -I/-L/-R for
> > prereqs and run cmake, to be able to build/install-to-destdir/test
> various
> > things.   These are often only 100 lines, 2/3 comments, and don't cause
> me
> > trouble."
> >
> > It sounds like your solution for generating up-to-date GDAL builds is
> > rather streamlined. Currently we are working on updating our current
> build
> > to GDAL 3.10.0. Were you to do the same how long would you estimate this
> > update would take? Also, is there any chance you could share any of your
> > scripts so I could compare them to ours?
>
> This is not baked, and not up to date.  For example it uses python 3.10
> for docs, and while pkgsrc can still build that, I'm on 3.12 and
> thinking about 3.13.  It is not up to date because I haven't had to run
> it lately.  I am the maintainer of gdal in pkgsrc, and I update there,
> and if that build works and passes tests, I don't need to build it
> outside of pkgsrc.   For a while I was chasing a few portability issues,
> and some floating point stuff, but now there's very little trouble.
>
> But, I am not statically linking, and I am living in a world where the
> dependencies all come from pkgsrc.  I typically build like this to check
> git master for portability bugs that have crept in (test ==, assuming
> things not required by POSIX, etc.) and to see if tests still pass (new
> tests are added, and floating point is hard :-).
>
> The last part is creating a distfile -- what the project would release
> so I can run that through packaging.
>
> Typically, for a new version, there's very little if anything I do.  I'd
> say if things build and test with no issues, it's 15 minutes of work.
> If there are build/test issues, those dominate the effort and updating
> the script is in the noise.
>
> (I have similar scripts for autoconf projects; same concept different
> spelling of build commands.)
>
> #!/bin/sh
>
> VERSION=3.6.0.dev0
>
> if [ -d $HOME/bin/ccache ]; then
>     echo "enabling ccache"
>     ccache -z
>     PATH=$HOME/bin/ccache:$PATH
> fi
>
> ##
>
> PREFIX=/usr/pkg
> LIBDIR=${PREFIX}/lib
>
> # In theory, BSD make is ok.
> MAKE=make
>
> # \todo Debug poppler includes not found.
>
> (rm -rf build destdir &&
>      mkdir build &&
>      cd build &&
>      cmake .. \
>            -LH \
>            -DCMAKE_COLOR_MAKEFILE=false \
>            -DCMAKE_INSTALL_PREFIX=${PREFIX} \
>            -DCMAKE_INSTALL_RPATH=${LIBDIR} \
>            > OUT.00.cmake 2>&1 &&
>      time ${MAKE} VERBOSE=1  > OUT.10.make 2>&1 &&
>      (time ${MAKE} check > OUT.20.check 2>&1 || true) && \
>      time ${MAKE} DESTDIR=../destdir install > OUT.30.install 2>&1 && \
>      echo BUILD DONE
> )
>
> ## Create a distfile for pkgsrc testing.
>
> # Set up tool variables.
> export MAKE=gmake
> export PYTHON=python3.10
> export SPHINXBUILD=sphinx-build-3.10
> # Remediate use of tools without search or variables.
> PATH=/home/gdt/bin/GDAL:$PATH
>
> (time ./mkgdaldist.sh ${VERSION} -url https://github.com/gdt/gdal >
> OUT.40.mkgdaldist 2>&1 &&
>      cp -p gdal-${VERSION}.tar.xz /usr/pkgsrc/distfiles &&
>      echo "COPIED DISTFILE")
> _______________________________________________
> gdal-dev mailing list
> gdal-dev at lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/gdal-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20250106/fdf1a4f2/attachment-0001.htm>