[geos-devel] dynamic_cast may yield null pointers in AbstractSTRtree::query and AbstractSTRtree::itemsTree on OS X

Fri Jan 5 10:36:03 PST 2018

AbstractSTRtree doesn't look particularly interesting at first glance.  I
only see a few linter issues like param names that don't match, a redundant
return on void method

:(

On Fri, Jan 5, 2018 at 10:04 AM, Sean Gillies <sean.gillies at gmail.com>
wrote:

> Hi Kurt,
>
> Thanks for the sympathy!
>
> It's the same library version. Same library build, even. I have a system
> that builds GEOS once, copies it into the two Python package trees, and
> uses install_name_tool to relink all the Python .so modules.
> https://github.com/matthew-brett/delocate (if you're curious) wraps both
> install_name_tool and otool and is what the SciPy team uses to make binary
> wheels (the Python distribution format) for numpy, scipy, pandas etc.
>
> $ delocate-listdeps --all ~/Downloads/scipy-1.0.0-cp27-c
> p27m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.
> macosx_10_10_intel.macosx_10_10_x86_64.whl
> /System/Library/Frameworks/Accelerate.framework/Versions/A/Accelerate
> /usr/lib/libSystem.B.dylib
> /usr/lib/libstdc++.6.0.9.dylib
> @loader_path/../../../../.dylibs/libgcc_s.1.dylib
> @loader_path/../../../../.dylibs/libgfortran.3.dylib
> @loader_path/../../../../.dylibs/libquadmath.0.dylib
> @loader_path/../../../.dylibs/libgcc_s.1.dylib
> @loader_path/../../../.dylibs/libgfortran.3.dylib
> @loader_path/../../../.dylibs/libquadmath.0.dylib
> @loader_path/../.dylibs/libgcc_s.1.dylib
> @loader_path/../.dylibs/libgfortran.3.dylib
> @loader_path/../.dylibs/libquadmath.0.dylib
> @loader_path/libgcc_s.1.dylib
> @loader_path/libquadmath.0.dylib
>
> This works like a charm for SciPy packages, which don't have overlapping
> dependencies like shapely and fiona do, and also works fine for shapely and
> fiona outside of the AbstractSTRTree module.
>
>
> On Thu, Jan 4, 2018 at 12:22 PM, Kurt Schwehr <schwehr at gmail.com> wrote:
>
>> Ouch :(
>>
>> Some thoughts... you might be getting different versions of the library
>> (e.g. different compiler versions or flags for the same version of
>> libgeos).  otool -L and dtrace are places to start, but aren't fun.  Is it
>> happening on multiple machines?  Using macports, fink, homebrew, or for the
>> libs?
>>
>> On Wed, Jan 3, 2018 at 10:34 AM, Sean Gillies <sean.gillies at gmail.com>
>> wrote:
>>
>>> Hi and Happy New Year, all.
>>>
>>> I found time recently to make notes on a frustrating issue that I'm
>>> having with GEOS in the Shapely binary wheels for OS X that I'm publishing
>>> on PyPI. I'm not certain whether there's a GEOS bug, a dynamic loader bug,
>>> a ctypes bug, misuse of computers on my part, or all four. I'd really
>>> appreciate a bit of advice from anyone with more C++ and Mach-O wisdom that
>>> would rule out a few of these possibilities.
>>>
>>> Here's the gist of it: Shapely uses dlopen (via Python's ctypes) to load
>>> libgeos_c.dylib at run time. The GEOS C++ library (libgeos-3.6.2.dylib, for
>>> example) is a dependent library of libgeos_c.dylib. I have been
>>> distributing the GEOS libs with my Shapely binaries for convenience of
>>> users, but Shapely also works with GEOS installed to the standard places on
>>> your system. I've never experienced or seen report of a problem with
>>> Shapely's loading of the GEOS libraries *in isolation* that hasn't been
>>> fixed.
>>>
>>> My Fiona package for Python also depends on libgeos_c.dylib, but in the
>>> more familiar way: it's a dependent library of GDAL, which is loaded when
>>> Fiona's C extension modules are loaded in Python.
>>>
>>> If we import fiona in a Python script and then import shapely from a
>>> binary wheel that includes GEOS libraries, the script will abort in either
>>> AbstractSTRtree::query or AbstractSTRtree::itemsTree because the dynamic
>>> casts yield null pointers. As far as I can tell, loading two copies of the
>>> C++ GEOS library is where the trouble starts. Is the trouble in the loader,
>>> the library code, or my builds? I do not know.
>>>
>>> There's no problem on Linux. The Linux library loader may be more
>>> foolproof or the library code might be compiled more correctly in a way
>>> that I don't yet see. I'm using the following flags in my builds – the dual
>>> architecture build is the only thing that seems unusual to me.
>>>
>>> environment =
>>>     MACOSX_DEPLOYMENT_TARGET=10.9
>>> configure-options =
>>>     CFLAGS="-arch i386 -arch x86_64 -O2 -Wl,-S -Wall -Wstrict-prototypes"
>>>     CXXFLAGS="-arch i386 -arch x86_64 -O2 -Wl,-S -Wall -Wstrict-prototypes"
>>>     LDFLAGS="-arch i386 -arch x86_64"
>>>
>>> The thing that makes me suspect that there is a localized bug in GEOS is
>>> that loading the library twice doesn't lead to failures in computing areas,
>>> lengths, predicates, or WKT serializations. Only in the AbstractSTRtree
>>> module as far as I can tell.
>>>
>>> I've made a ticket at https://trac.osgeo.org/geos/ticket/848 and have
>>> attached a script that reproduces the problem as well as demonstrating that
>>> other GEOS modules are unaffected.
>>>
>>> It's interesting, but mostly baffling and humbling to go so deep into
>>> the weeds of dynamic library loading. I'm over my head here and super
>>> grateful for insights and discussion.
>>>
>>> --
>>> Sean Gillies
>>>
>>> _______________________________________________
>>> geos-devel mailing list
>>> geos-devel at lists.osgeo.org
>>> https://lists.osgeo.org/mailman/listinfo/geos-devel
>>>
>>
>>
>>
>> --
>> --
>> http://schwehr.org
>>
>> _______________________________________________
>> geos-devel mailing list
>> geos-devel at lists.osgeo.org
>> https://lists.osgeo.org/mailman/listinfo/geos-devel
>>
>
>
>
> --
> Sean Gillies
>
> _______________________________________________
> geos-devel mailing list
> geos-devel at lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/geos-devel
>

-- 
--
http://schwehr.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/geos-devel/attachments/20180105/8f91f794/attachment-0001.html>