[gdal-dev] GDAL Seg Fault - for info

[Even Rouault]

I've come up with a quicky&dirty script 
https://github.com/rouault/symbol_clash_detector/blob/master/symbol_clash_detector.py 
that works only for Linux and attempts to find C symbol clashes in the 
.so files loaded in a process (C++ is trickier because due to template & 
inline code, functions from the standard library are duplicated in C++ 
libraries)

When run on Conda gdal-3.6.0, it outputs:

$ PYTHONPATH=/home/even/symbol_clash_detector python -c "from osgeo 
import gdal; import symbol_clash_detector"
WARNING: libVersionPoint present in 
/home/even/miniconda3/envs/test/lib/libplc4.so is also present in 
/home/even/miniconda3/envs/test/lib/libnspr4.so
WARNING: locale_charset present in 
/home/even/miniconda3/envs/test/lib/libiconv.so.2.6.1 is also present in 
/home/even/miniconda3/envs/test/lib/libcharset.so.1.0.0
WARNING: libVersionPoint present in 
/home/even/miniconda3/envs/test/lib/libplds4.so is also present in 
/home/even/miniconda3/envs/test/lib/libnspr4.so

I'm not knowledgeable enough to assess if those 3 instances are 
problematic (symbols with same name and content are OK), hopefully not, 
as there are more or less system libraries.

When run on my dev GDAL build with drivers using proprietary SDKs, it 
does find real symbol clashes that I was aware of, due to those SDKs 
embedding symbol libraries (like libxml2 in FileGDB SDK, lcms in ECW 
SDK, ...)

Le 21/11/2022 à 13:37, Even Rouault a écrit :
>
> Paul,
>
> Looking at 
> https://stackoverflow.com/questions/63161532/python-segmentation-fault-when-using-zipfile-while-gdal-package-loaded
>
> shows an interesting thing in a gdb backtrace
>
> |#0 0x00007ffff7e3aa50 in free () from /usr/lib/libc.so.6 #1 
> 0x00007ffff485ae0f in inflateReset2 () from /usr/lib/libcfitsio.so.9 
> #2 0x00007ffff39ec1a4 in inflateInit2_ () from /usr/lib/libz.so.1 |
> ||
>
> One can see that libz inflateInit2_() ends up calling libcfitsio's 
> inflateReset2(), but libcfitsio is not a dependency of libz ! That's 
> the reverse. Which suggests that some builds of libcfitsio use an 
> internal outdated copy of libz, without symbol renaming. Looking at 
> conda-forge cfitsio recipe I see 
> https://github.com/conda-forge/cfitsio-feedstock/blob/main/recipe/unvendor_zlib.patch 
> (added per https://github.com/conda-forge/cfitsio-feedstock/pull/22) 
> which changes cfitsio to use conda-forge zlib, to avoid such issue.
>
> With latest conda-forge gdal, requests, openpyxl, I can't trigger 
> crashes with the reproducers of the various mentioned tickets.
>
> I'd suggest you run your code under gdb or Valgrind and look if there 
> are not funny symbol calls like the above.
>
> Also if use pip module, avoid using their binary wheels. For example 
> to make sure that pyproj's embedded PROJ doesn't conflict with the 
> PROJ used by GDAL, to avoid similar issues where you'd have 2 versions 
> of the same lib in the same process.
>
> Even
>
> Le 21/11/2022 à 12:40, Paul Harwood a écrit :
>> This may not be a GDAL bug and as such I have not raised it as an 
>> issue but I wanted to put it here for anyone else who comes across 
>> the same problem.
>>
>> There does appear to be some interference between, at least the conda 
>> distribution of, GDAL and something else within Python causing a 
>> segmentation fault.
>>
>> ----------------------------------------
>> SYMPTOMS
>>
>> I have a Python application that uses GDAL (and PDAL Python and MDAL 
>> Python and PyProj) wrapped in Kivy - so it is relatively complicated.
>>
>> I was just clearing tech debt / updating etc. GDAL to 3.6 but also 
>> the other apps to latest versions (including kivy).
>>
>> BEFORE - if I tried to open an unrecognised file in GDAL I got an 
>> Exception (using UseExceptions() ). This is related to Python Duck Typing
>>
>> AFTER - I get a seg fault and no debug info.
>>
>> This happened in exactly the same way on two machines - one Mac and 
>> one Windows - so I don't  think it is a bad config. Attempts to 
>> create a minimum example do not work - suggesting that it is a 
>> complicated interaction between the various components
>>
>> I am using conda as the environment manager.
>>
>> ------------------------------------------
>>
>> ULTIMATE(-ish) SOLUTION
>>
>> After getting some diagnostic data and some experimentation, it 
>> turned out that the fault is coming in the error handling. I managed 
>> to avoid the segmentation fault by creating and registering a 
>> dedicated error handler for GDAL.
>>
>> --------------------------------------------
>> DEEPER DISCUSSION
>>
>> I found some discussion from a couple of years ago about a problem 
>> that sounds similar (see 
>> https://github.com/conda-forge/gdal-feedstock/issues/365) - it was 
>> not solved but was not very reproducible so was closed.
>>
>> This suggests an interference between requests and gdal - depending 
>> on the order in which they are imported. Certainly - kivy (or more 
>> precisely kivyMD) is using requests but I was not able to solve the 
>> problem by changing import orders (kivy has a lot of asynch and event 
>> based things happening and the app is very modular so it is not that 
>> simple).
>>
>> However - if the interference is in the error handler - then that 
>> would explain why it was difficult to reproduce (need to hit the 
>> right exception in the right way).
>>
>> Not sure if we can fix this one - but if anyone else does come across 
>> a similar problem I would suggest to try and create a custom error 
>> handler to see if the problem is there.
>>
>> _______________________________________________
>> gdal-dev mailing list
>> gdal-dev at lists.osgeo.org
>> https://lists.osgeo.org/mailman/listinfo/gdal-dev
> -- 
> http://www.spatialys.com
> My software is free, but my time generally not.
>
> _______________________________________________
> gdal-dev mailing list
> gdal-dev at lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/gdal-dev

-- 
http://www.spatialys.com
My software is free, but my time generally not.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20221122/df78cb7e/attachment.htm>