[gdal-dev] Improving details of the project build system and/or documentation

[Even Rouault]

> Now that the project has made it to CMake, I wonder if there is enthusiasm to break apart GDAL's build system to take advantage of its plugin capability in a default fashion to cut down on the massiveness of our default stuff-you-actually-want build situation. People rightly complain that libgdal.so links too much stuff, and in most cases, people don't need it all.
>
> What if we were to default to having drivers be free-standing CMake projects that installed versioned plugin shared objects into known locations that could be activated at library load (or use) time? GDAL has this dynamic plugin capability right now, but it is not widely used, and build configuration is not so convenient for its usage. Plugins would allow each of these drivers to link to their esoteric dependencies without polluting the common library, and the situation we have now, where multiple drivers have similar sets of dependencies, could be handled in some kind of hierarchical fashion based on common dependencies.
>
> For deployments like rasterio's wheels, external GDAL plugins could be loaded at runtime by the rasterio libgdal.so, and rasterio wouldn't need to be responsible for building a libgdal.so that provided kitchen sink capabilities. People could show up with their own driver plugins that included dependencies as needed.
>
> GDAL's reputation of being hard to build and deploy is well earned and deserved. IMO, the CMake configuration has increased its build speed, system flexibility, and commonality on supported platforms, but it hasn't addressed a core challenge which is GDAL links to the world one DLL at a time. I don't know if the project wants to prioritize this topic relative to other issues, but we have some tools and resources available to us to improve the situation if so.

Some thoughts/questions regarding this:

- you definitely don't want to build all drivers (that can be built as 
plugins) as plugins. This works (this is actually my dev setup, to 
ensure that the core exports all the symbols needed for drivers built as 
plugins), but I've found there's a measurable penalty in doing so (300 
ms to load the 127 plugins on my system, ie each GDAL command line 
invokation will at least take 300 ms, which might be a significant 
penalty in some workflows) . So GDAL_ENABLE_PLUGINS_NO_DEPS=YES (cf 
https://gdal.org/development/building_from_source.html#cmdoption-arg-GDAL_ENABLE_PLUGINS_NO_DEPS-BOOL) 
is typically discouraged.  But 10 or so plugins should be fine.

- you can "emulate" building a single driver as plugin, by re-running a 
GDAL build with just the dependencies of that driver, and setting 
-DGDAL_ENABLE_DRIVER_xxxx_PLUGIN=ON. The only build artifact of interest 
in that use case is the [gdal|ogr]_XXX.so/dll corresponding to that 
driver. The main inconvenience I can see with this current approach is 
that you have to pay the price for the core lib to be rebuilt, whereas 
with the free-standing CMake project, you'd point to the installed GDAL 
headers & lib

- by free-standing CMake project, what do you mean ? Couldn't that just 
the existing frmts/XXXX/CMakeLists.txt file that would detect it is 
called as the top level CMakeLists.txt (probably by checking 
${CMAKE_CURRENT_SOURCE_DIR} == ${PROJECT_SOURCE_DIR}) and then change 
its logic to find libgdal and its dependencies ? And users would "cmake 
-S frmts/XXXX -B plugin_XXXX" to configure just for that plugin.

- I don't understand the "where multiple drivers have similar sets of 
dependencies, could be handled in some kind of hierarchical fashion 
based on common dependencies". why is a special case needed if several 
drivers share the same dependencies ? Or maybe you were thinking to 
drivers depending on other drivers ? That situation can indeed happen 
currently, but when it happens, such drivers aren't allowed to be built 
as plugins. They have to be built with libgdal. Both because it makes 
the build system simpler, and also to avoid deployment complication 
where you'd have to know that gdal_XXXX.so links against gdal_YYYY.so.

- we would need to keep the current monolithic CMake project approach 
working, at the very least for people doing static builds, but even for 
people doing dynamic builds as it might be still be more practical if 
you can install all the dependencies at once. Having the standalone 
capability in addition would be an extra complexity, so we need to be 
sure it addresses real use cases, and perhaps restrict it to a few 
select drivers. Obvious candidates for standalone CMake projects would 
likely be for the few drivers that have proprietary dependencies (ECW, 
JP2KAK, MrSID, Oracle, ...) and aren't shipped by FOSS binary distributions.

- offering the possibility of building just a driver as a plugin in a 
unit way would require people to build against the same GDAL headers as 
libgdal. So people would have to download the same version of the GDAL 
sources as the one used by the libgdal provided by rasterio + having the 
required SDK + installing the libs of the SDK in a path where 
Python/rasterio can find them. Is it in the expected cultural background 
of rasterio users to do that on their own ?

Even

-- 

http://www.spatialys.com
My software is free, but my time generally not.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20230530/c60ca5c9/attachment.htm>