[osgeo4w-dev] [osgeo4w] #692: Python 3.9 on v2 installer breaks part of stdlib

[OSGeo4W]

#692: Python 3.9 on v2 installer breaks part of stdlib
-----------------------+---------------------------
Reporter:  akominlsfi  |      Owner:  osgeo4w-dev@…
    Type:  defect      |     Status:  new
Priority:  major       |  Component:  Installer
 Version:              |   Keywords:
-----------------------+---------------------------
 Python 3.9 (with v2 installer) breaks stdlib usage: ssl and sqlite, or
 venv, depending on the way python is executed.

 Our use case currently is to create a venv for QGIS plugin development on
 Windows machines, using `python-qgis-ltr -m venv --system-site-packages`,
 an additional .pth file in the created venv pointing to qgis python module
 directory (since those are not in site-packages) and setting the necessary
 dll directories from osgeo install to PATH. That allows installing
 development modules to the virtual environment and keeping osgeo install
 site packages clean.

 For current v2 qgis-ltr package the python-qgis-ltr.bat points to the exe
 in `apps/python39/python.exe`, ssl and sqlite dlls are in `bin`, and
 neither of those can be resolved by default. Creating a venv works, but it
 has the same problem for ssl and sqlite.

 When
 [https://github.com/jef-n/OSGeo4W/commit/6f17d96ce12e8f7d47bc5f909a07278fb151c3d5
 this fix] is made, the bat will point to the exe in `bin/python.exe` and
 ssl and sqlite dlls can be resolved by default. This will however break
 the venv module, since the `PYTHONHOME` that is setup by the bat cannot be
 baked into the venv, and the venv will actually try to find the standard
 library paths relative the original executable, eg. `bin/DLLS`, `bin/lib`.

 It seems previously (in v1) no dlls were duplicated to
 `apps/python37/dlls` since setting `bin` to PATH was enough to resolve
 sqlite and sll dlls. Current v2 installer duplicates only the ssl dlls to
 the `apps/python39/dlls` directory, but there seems to be a problem:
 libcrypto-1_1.dll and libssl-1_1.dll are found next to the _ssl.pyd
 module, but it actually seems to fail due to libssl-1_1.dll having a
 dependency on another dll found only in `bin`, libcrypto-1_1_x64.dll.
 Copying that to `apps/python39/dlls`, using the original bat pointing to
 the exe in `apps/python39/python.exe` and starting an interactive console
 `import ssl` actually now succeeds. Same thing with the sqlite3 module,
 `import sqlite3` works after copying sqlite3.dll from `bin` to
 `apps/python39/dlls`. There is
 [https://github.com/jef-n/OSGeo4W/commit/547e0bdbd3a640e4b92dd38cefea95e2c83ab5ef
 #diff-dc4b1e47c00b8f477e2a83335db08806820d54063a982eedf8c4ac205ec7ac88
 this change] on how ssl dlls work, but seems the problem with x64 vs
 non-x64 named dlls has resurfaced?

 Some of this has been also documented on
 [https://github.com/qgis/QGIS/issues/44185 this QGIS issue] when I looked
 into this previously.

 Is the duplication of stdlib required dlls also next to the relevant .pyd
 modules a valid fix? Since ssl dlls are duplicated already (partially,
 x64-labeled missing), sqlite dll is the only one I found relevant by
 looking at the .pyd files with Dependency Walker. This would help
 somewhat, but still leave a problem with all the osgeo-installed python
 modules that have their dlls are `bin` and .pyds somewhere in `site-
 packages`. The bat fix pointing to the different exe works for those, but
 breaks the venv module.

 Another way I thought just now could be to patch the osgeo-installed
 python standard library importlib module? It could be possible to add all
 the dll paths of the osgeo install tree in importlib init. This is
 currently done by qgis and PyQt5 modules to find their own requirements.
 This could help to preserve the original bat style (pointing to the
 `apps/python39/python.exe`), no duplication necessary for the stdlib dlls,
 and `import ssl`/`import sqlite3` would work due to the patch, and venv
 would work since the original executable directory structure is as
 expected (and the importlib patch is also valid for imports in the venv
 since that comes from the shared stdlib).

-- 
Ticket URL: <https://trac.osgeo.org/osgeo4w/ticket/692>
OSGeo4W <http://trac.osgeo.org/osgeo4w>
OSGeo4W is the Windows installer and package environment for the OSGeo stack.