[gdal-dev] Python Wheels for gdal

[Jason Roberts]

As a 15-year user of GDAL's Python bindings and a long time lurker on this list, I just wanted to thank all of you who are contemplating the Python packaging problem (and thank you for GDAL in general). I maintain a small package of Python-based utilities used by marine ecologists, mainly in an academic research setting, usually accessed from ArcGIS though GUI wrappers. Installation is one of the biggest obstacles for these users. Anything you do to further improve it (e.g. pip install gdal) would be very much appreciated.

I also wanted to beg that you include Windows in your contemplations. On Windows, there is the classic "DLL Hell" problem similar to the symbol clashing problem you've been discussing for Linux. You guys are probably more familiar with it than me at this point, but the basic issue used to be that on Windows, a given process could only have one DLL with a given file name loaded. Because GDAL dynamically links to so many libraries, it can be a problem to access it from complicated programs, such as ArcGIS (which has its own private copy of GDAL), or in concert with other libraries that might have common dependencies (e.g. on netCDF, JPEG, etc.), at least when all of the DLLs have the same names. Anyway, this stuff seems somewhat tangential to the Python packaging problem, but any improvements you can make in to this stuff is also very appreciated.

Thanks again and all the best,

Jason

> -----Original Message-----
> From: gdal-dev <gdal-dev-bounces at lists.osgeo.org> On Behalf Of Even
> Rouault
> Sent: Friday, January 31, 2020 9:53 AM
> To: gdal-dev at lists.osgeo.org
> Subject: Re: [gdal-dev] Python Wheels for gdal
> 
> On vendredi 31 janvier 2020 15:14:39 CET Mateusz Loskot wrote:
> > On Fri, 31 Jan 2020 at 10:58, Christoph Paulik <cpaulik at vandersat.com>
> wrote:
> > > My initial motivation was that I would like `pip install gdal` to
> > > just work.
> > I do share your view.
> > The pip install is always the canonical Pythonic way for me and I'd be
> > very happy myself to be able to install GDAL that way.
> >
> > > I guess that it is unrealistic to fix any of that, so let's get back
> > > on topic.
> > My point is that, IMO, such initiative is best to be run as a project
> > independent from GDAL development, outside GDAL repository, for
> > practical reasons.
> 
> Except that if we wanted "pip install gdal" to include wheels, that should
> be done as part of the gdal Pyython package that is managed in swig/python
> of GDAL repository. Something outside should use another package name
> 
> <brainstorming>
> Thinking about Sean's very valid point about wheels and symbol clashes
> when loading different version of the same library (or perhaps even the
> same version twice), I'm wondering if using symbol versioning, at least on
> Linux, wouldn't solve this. Imagine that you'd recompile GDAL and all its
> dependencies such that all exported symbols (functions & global
> variables), with some "gdal_wheel" marker, then those libraries would be
> effectively private to the Python package. I know that at some past point
> (GDAL 1.8 I think), Debian packaged GDAL with versionned symbols (from a
> distribution point of view, versionned symbols aren't necessarily that
> great from what I read, but for the use case we discuss here, that could
> perhaps be an option).
> 
> Hopefully, there would be some existing tools to automate this. At linking
> time. Or perhaps as a post processing stage, messing directly with the ELF
> format.
> 
> I could imagine thought that symbol renaming/versionning wouldn't fly very
> well with dlopen()/dlsym() use that libraries could make. But I don't
> think this is extensive used in the GDAL use case. On top of my head, a
> few cases might be:
> * typically the old way GDAL loaded PROJ4. But no longer an issue with
> GDAL 3, or can be avoided with GDAL 2.x as well when building with --with-
> [static-]proj[4])
> * the OGDI library loading its plugin. But OGDI is sufficiently a odd
> beast that we don't need/want this in a general purpose wheel.
> </brainstorming>
> 
> Even
> 
> --
> Spatialys - Geospatial professional services
> https://urldefense.proofpoint.com/v2/url?u=http-
> 3A__www.spatialys.com&d=DwIGaQ&c=imBPVzF25OnBgGmVOlcsiEgHoG1i6YHLR0Sj_gZ4a
> dc&r=cJfJ4ejc1xbg_qb47Pg1OoRq1GfFGvWbDD2PT7-
> fBKk&m=aoKWdL0rF70E3ttLt_aZ96nhL_Cdj1rsNXRsMBVM_9M&s=7T6raBfaPIXagvEPCCZDh
> nZMGlqHNWWjv1rdKxYLKEw&e=
> _______________________________________________
> gdal-dev mailing list
> gdal-dev at lists.osgeo.org
> https://urldefense.proofpoint.com/v2/url?u=https-
> 3A__lists.osgeo.org_mailman_listinfo_gdal-
> 2Ddev&d=DwIGaQ&c=imBPVzF25OnBgGmVOlcsiEgHoG1i6YHLR0Sj_gZ4adc&r=cJfJ4ejc1xb
> g_qb47Pg1OoRq1GfFGvWbDD2PT7-
> fBKk&m=aoKWdL0rF70E3ttLt_aZ96nhL_Cdj1rsNXRsMBVM_9M&s=Hgl9GE67s6g8AcPh3VnFC
> QzQkhgnjRBdszdhn338ddM&e=