[geos-devel] dynamic_cast may yield null pointers in AbstractSTRtree::query and AbstractSTRtree::itemsTree on OS X

[Sean Gillies]

Hi Kurt,

Thanks for the sympathy!

It's the same library version. Same library build, even. I have a system
that builds GEOS once, copies it into the two Python package trees, and
uses install_name_tool to relink all the Python .so modules.
https://github.com/matthew-brett/delocate (if you're curious) wraps both
install_name_tool and otool and is what the SciPy team uses to make binary
wheels (the Python distribution format) for numpy, scipy, pandas etc.

$ delocate-listdeps --all ~/Downloads/scipy-1.0.0-cp27-
cp27m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_
x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl
/System/Library/Frameworks/Accelerate.framework/Versions/A/Accelerate
/usr/lib/libSystem.B.dylib
/usr/lib/libstdc++.6.0.9.dylib
@loader_path/../../../../.dylibs/libgcc_s.1.dylib
@loader_path/../../../../.dylibs/libgfortran.3.dylib
@loader_path/../../../../.dylibs/libquadmath.0.dylib
@loader_path/../../../.dylibs/libgcc_s.1.dylib
@loader_path/../../../.dylibs/libgfortran.3.dylib
@loader_path/../../../.dylibs/libquadmath.0.dylib
@loader_path/../.dylibs/libgcc_s.1.dylib
@loader_path/../.dylibs/libgfortran.3.dylib
@loader_path/../.dylibs/libquadmath.0.dylib
@loader_path/libgcc_s.1.dylib
@loader_path/libquadmath.0.dylib

This works like a charm for SciPy packages, which don't have overlapping
dependencies like shapely and fiona do, and also works fine for shapely and
fiona outside of the AbstractSTRTree module.


On Thu, Jan 4, 2018 at 12:22 PM, Kurt Schwehr <schwehr at gmail.com> wrote:

> Ouch :(
>
> Some thoughts... you might be getting different versions of the library
> (e.g. different compiler versions or flags for the same version of
> libgeos).  otool -L and dtrace are places to start, but aren't fun.  Is it
> happening on multiple machines?  Using macports, fink, homebrew, or for the
> libs?
>
> On Wed, Jan 3, 2018 at 10:34 AM, Sean Gillies <sean.gillies at gmail.com>
> wrote:
>
>> Hi and Happy New Year, all.
>>
>> I found time recently to make notes on a frustrating issue that I'm
>> having with GEOS in the Shapely binary wheels for OS X that I'm publishing
>> on PyPI. I'm not certain whether there's a GEOS bug, a dynamic loader bug,
>> a ctypes bug, misuse of computers on my part, or all four. I'd really
>> appreciate a bit of advice from anyone with more C++ and Mach-O wisdom that
>> would rule out a few of these possibilities.
>>
>> Here's the gist of it: Shapely uses dlopen (via Python's ctypes) to load
>> libgeos_c.dylib at run time. The GEOS C++ library (libgeos-3.6.2.dylib, for
>> example) is a dependent library of libgeos_c.dylib. I have been
>> distributing the GEOS libs with my Shapely binaries for convenience of
>> users, but Shapely also works with GEOS installed to the standard places on
>> your system. I've never experienced or seen report of a problem with
>> Shapely's loading of the GEOS libraries *in isolation* that hasn't been
>> fixed.
>>
>> My Fiona package for Python also depends on libgeos_c.dylib, but in the
>> more familiar way: it's a dependent library of GDAL, which is loaded when
>> Fiona's C extension modules are loaded in Python.
>>
>> If we import fiona in a Python script and then import shapely from a
>> binary wheel that includes GEOS libraries, the script will abort in either
>> AbstractSTRtree::query or AbstractSTRtree::itemsTree because the dynamic
>> casts yield null pointers. As far as I can tell, loading two copies of the
>> C++ GEOS library is where the trouble starts. Is the trouble in the loader,
>> the library code, or my builds? I do not know.
>>
>> There's no problem on Linux. The Linux library loader may be more
>> foolproof or the library code might be compiled more correctly in a way
>> that I don't yet see. I'm using the following flags in my builds – the dual
>> architecture build is the only thing that seems unusual to me.
>>
>> environment =
>>     MACOSX_DEPLOYMENT_TARGET=10.9
>> configure-options =
>>     CFLAGS="-arch i386 -arch x86_64 -O2 -Wl,-S -Wall -Wstrict-prototypes"
>>     CXXFLAGS="-arch i386 -arch x86_64 -O2 -Wl,-S -Wall -Wstrict-prototypes"
>>     LDFLAGS="-arch i386 -arch x86_64"
>>
>> The thing that makes me suspect that there is a localized bug in GEOS is
>> that loading the library twice doesn't lead to failures in computing areas,
>> lengths, predicates, or WKT serializations. Only in the AbstractSTRtree
>> module as far as I can tell.
>>
>> I've made a ticket at https://trac.osgeo.org/geos/ticket/848 and have
>> attached a script that reproduces the problem as well as demonstrating that
>> other GEOS modules are unaffected.
>>
>> It's interesting, but mostly baffling and humbling to go so deep into the
>> weeds of dynamic library loading. I'm over my head here and super grateful
>> for insights and discussion.
>>
>> --
>> Sean Gillies
>>
>> _______________________________________________
>> geos-devel mailing list
>> geos-devel at lists.osgeo.org
>> https://lists.osgeo.org/mailman/listinfo/geos-devel
>>
>
>
>
> --
> --
> http://schwehr.org
>
> _______________________________________________
> geos-devel mailing list
> geos-devel at lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/geos-devel
>



-- 
Sean Gillies
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/geos-devel/attachments/20180105/ce5e03d6/attachment.html>