[osgeo4w-dev] [osgeo4w] #692: Python 3.9 on v2 installer breaks part of stdlib
trac_osgeo4w at osgeo.org
Wed Sep 29 00:21:56 PDT 2021
#692: Python 3.9 on v2 installer breaks part of stdlib
Reporter: akominlsfi | Owner: osgeo4w-dev@…
Type: defect | Status: new
Priority: major | Component: Installer
Version: | Keywords:
Python 3.9 (with v2 installer) breaks stdlib usage: ssl and sqlite, or
venv, depending on the way python is executed.
Our use case currently is to create a venv for QGIS plugin development on
Windows machines, using `python-qgis-ltr -m venv --system-site-packages`,
an additional .pth file in the created venv pointing to qgis python module
directory (since those are not in site-packages) and setting the necessary
dll directories from osgeo install to PATH. That allows installing
development modules to the virtual environment and keeping osgeo install
site packages clean.
For current v2 qgis-ltr package the python-qgis-ltr.bat points to the exe
in `apps/python39/python.exe`, ssl and sqlite dlls are in `bin`, and
neither of those can be resolved by default. Creating a venv works, but it
has the same problem for ssl and sqlite.
this fix] is made, the bat will point to the exe in `bin/python.exe` and
ssl and sqlite dlls can be resolved by default. This will however break
the venv module, since the `PYTHONHOME` that is setup by the bat cannot be
baked into the venv, and the venv will actually try to find the standard
library paths relative the original executable, eg. `bin/DLLS`, `bin/lib`.
It seems previously (in v1) no dlls were duplicated to
`apps/python37/dlls` since setting `bin` to PATH was enough to resolve
sqlite and sll dlls. Current v2 installer duplicates only the ssl dlls to
the `apps/python39/dlls` directory, but there seems to be a problem:
libcrypto-1_1.dll and libssl-1_1.dll are found next to the _ssl.pyd
module, but it actually seems to fail due to libssl-1_1.dll having a
dependency on another dll found only in `bin`, libcrypto-1_1_x64.dll.
Copying that to `apps/python39/dlls`, using the original bat pointing to
the exe in `apps/python39/python.exe` and starting an interactive console
`import ssl` actually now succeeds. Same thing with the sqlite3 module,
`import sqlite3` works after copying sqlite3.dll from `bin` to
`apps/python39/dlls`. There is
this change] on how ssl dlls work, but seems the problem with x64 vs
non-x64 named dlls has resurfaced?
Some of this has been also documented on
[https://github.com/qgis/QGIS/issues/44185 this QGIS issue] when I looked
into this previously.
Is the duplication of stdlib required dlls also next to the relevant .pyd
modules a valid fix? Since ssl dlls are duplicated already (partially,
x64-labeled missing), sqlite dll is the only one I found relevant by
looking at the .pyd files with Dependency Walker. This would help
somewhat, but still leave a problem with all the osgeo-installed python
modules that have their dlls are `bin` and .pyds somewhere in `site-
packages`. The bat fix pointing to the different exe works for those, but
breaks the venv module.
Another way I thought just now could be to patch the osgeo-installed
python standard library importlib module? It could be possible to add all
the dll paths of the osgeo install tree in importlib init. This is
currently done by qgis and PyQt5 modules to find their own requirements.
This could help to preserve the original bat style (pointing to the
`apps/python39/python.exe`), no duplication necessary for the stdlib dlls,
and `import ssl`/`import sqlite3` would work due to the patch, and venv
would work since the original executable directory structure is as
expected (and the importlib patch is also valid for imports in the venv
since that comes from the shared stdlib).
Ticket URL: <https://trac.osgeo.org/osgeo4w/ticket/692>
OSGeo4W is the Windows installer and package environment for the OSGeo stack.
More information about the osgeo4w-dev