[gdal-dev] Improving details of the project build system and/or documentation

[Howard Butler]

> On May 28, 2023, at 5:20 PM, Sean Gillies <sean.gillies at gmail.com> wrote:
> 
> Testing build systems is tough. I know GDAL can't test that all optional drivers can be configured and built in isolation. At the same time, profiles of drivers between the bare minimum and everything-but-the-kitchen-sink are useful to some communities of users, like the projects that would like to "pip install rasterio" in their own builds. I resented being told on this list (not by you, Even) that I could take it or leave it with regards to the current build situation. For my part, I can do a better job of trying rasterio wheel builds against the upcoming versions of GDAL, and I will do so.

Now that the project has made it to CMake, I wonder if there is enthusiasm to break apart GDAL's build system to take advantage of its plugin capability in a default fashion to cut down on the massiveness of our default stuff-you-actually-want build situation. People rightly complain that libgdal.so links too much stuff, and in most cases, people don't need it all.  

What if we were to default to having drivers be free-standing CMake projects that installed versioned plugin shared objects into known locations that could be activated at library load (or use) time? GDAL has this dynamic plugin capability right now, but it is not widely used, and build configuration is not so convenient for its usage. Plugins would allow each of these drivers to link to their esoteric dependencies without polluting the common library, and the situation we have now, where multiple drivers have similar sets of dependencies, could be handled in some kind of hierarchical fashion based on common dependencies. 

For deployments like rasterio's wheels, external GDAL plugins could be loaded at runtime by the rasterio libgdal.so, and rasterio wouldn't need to be responsible for building a libgdal.so that provided kitchen sink capabilities. People could show up with their own driver plugins that included dependencies as needed.

GDAL's reputation of being hard to build and deploy is well earned and deserved. IMO, the CMake configuration has increased its build speed, system flexibility, and commonality on supported platforms, but it hasn't addressed a core challenge which is GDAL links to the world one DLL at a time. I don't know if the project wants to prioritize this topic relative to other issues, but we have some tools and resources available to us to improve the situation if so.

Howard