[Qgis-user] [Caution - External w/link] Re: Troubleshooting Python Script - QGIS library Import on Recent Versions on Mac OS X

Tue Oct 6 14:25:14 PDT 2026

Dear Greg and List,

So, here is another brick in the wall of our understanding. I am continuing
to investigate two paths to be able to run external Python programs using
QGIS.

1. Creating a Python environment from the python3.12 included with QGIS 4.x
(now 4.2.3).
2. Building QGIS myself on my own machine and using the included Python
environment from my build.

Both have roadblocks, but we may be a few more feet down the path.

Regarding APPROACH #1, I can do
/Applications/QGIS-final-4_2_3.app/Contents/MacOS/python -m venv
--without-pip /path/to/my/venv
And I get a venv. Note that trying to install pip with this command makes
it fail, hence the --without-pip flag.

Well, I need pip. At the very least I need to be able to install debugpy.
So then after activating the venv I try:
python3.12 -m ensurepip --upgrade

And isn't that interesting -- it fails but I get some good output.
---------
Could not find platform independent libraries <prefix>
Could not find platform dependent libraries <exec_prefix>
Python path configuration:
  PYTHONHOME = (not set)
  PYTHONPATH = (not set)
  program name = 'python3.12'
  isolated = 0
  environment = 1
  user site = 1
  safe_path = 0
  import site = 1
  is in build tree = 0
  stdlib dir =
'/Users/runner/work/QGIS/QGIS/build/vcpkg_installed/arm64-osx-dynamic-release/lib/python3.12'
  sys._base_executable =
'/Applications/QGIS-final-4_2_3.app/Contents/MacOS/python3.12'
  sys.base_prefix =
'/Users/runner/work/QGIS/QGIS/build/vcpkg_installed/arm64-osx-dynamic-release'
  sys.base_exec_prefix =
'/Users/runner/work/QGIS/QGIS/build/vcpkg_installed/arm64-osx-dynamic-release'
  sys.platlibdir = 'lib'
  sys.executable = '/path/to/my/venv/bin/python3.12'
  sys.prefix =
'/Users/runner/work/QGIS/QGIS/build/vcpkg_installed/arm64-osx-dynamic-release'
  sys.exec_prefix =
'/Users/runner/work/QGIS/QGIS/build/vcpkg_installed/arm64-osx-dynamic-release'
  sys.path = [

'/Users/runner/work/QGIS/QGIS/build/vcpkg_installed/arm64-osx-dynamic-release/lib/python312.zip',

'/Users/runner/work/QGIS/QGIS/build/vcpkg_installed/arm64-osx-dynamic-release/lib/python3.12',

'/Users/runner/work/QGIS/QGIS/build/vcpkg_installed/arm64-osx-dynamic-release/lib/python3.12/lib-dynload',
  ]
Fatal Python error: init_fs_encoding: failed to get the Python codec of the
filesystem encoding
Python runtime state: core initialized
ModuleNotFoundError: No module named 'encodings'

Current thread 0x0000000202c63f80 (most recent call first):
  <no Python frame>
---------

Okay, so now I see things a bit more clearly. The Mac where QGIS is built
for binary distribution has a user called "runner" where all the automated
magic happens. Just about everything I would want to change in either the
Python environment or the dylibs probably points to "/Users/runner/work" or
one of its subdirectories.

I don't know enough about Python builds to know if I can change all the
sys. values above and maybe succeed in adding pip. I would love some
insight as to whether it can be done.

So, this brings us back to APPROACH #2 where I build everything on my own
machine. I think I know where my problem is there.

When I last wrote in, I would fail to build QGIS 4.x on my own machine
during the building of py-rpds. I wrote up a bug for them and did some more
digging in logs and found that the deeper cause of the build errors
included the phrase "mis-aligned LINKEDIT". It turned out quite a few
builds involving Rust are having similar issues on OS X. To quote one
submission, "macOS 27's dyld now requires 8-byte alignment on the Mach-O
LINKEDIT string table" which in turn means you have to build with newer
versions of Apple's tools which can pad the table to 8-bytes.

It appears that various Rust projects are now patching to get around this
issue, but their patches are fairly recent and have not made their way into
QGIS builds yet. The two examples I put into comments on my vcpkg issue at
https://github.com/Microsoft/vcpkg/issues/54095 cite some fixes that have
been entered.

I wonder if this is the sort of thing I should file an issue with QGIS
about, or whether it will resolve on its own during a refresh of the build
projects QGIS uses when building with vcpkg on the Mac?

Anyway, that's my progress over the last week or so of intermittently
investigating all this. Once a resolution comes one way or another I'm
nailing my config down and freezing it in nitrogen.

On a side note, when I read the blog post stating that the next LTR would
be 4.4 as they were still squashing a lot of transition issues, I felt seen.

Kind Regards,
Johnnie

On Wed, Sep 23, 2026 at 6:32 PM Greg Troxel <gdt at lexort.com> wrote:

> "Odom, Johnnie" <JOdom at ecsdfl.us> writes:
>
> > Thank you for all of your advice and help. I've been intermittently
> trying
> > to troubleshoot ever since. I'm writing for anyone who comes across this
> > thread later, and to throw out a few additional possibilities for group
> > discussion.
>
> There's a lot in there and I think this digging in and documenting is
> helpful.
>
> > On the Mac, the equivalent of ldd is "otool -L".
> > Without explaining the rabbit holes I have been down the last several
> > weeks, I can simply say that the dylib situation on OS X is complicated.
>
> Agreed; dylib is more complicated and less understood than ELF libraries
> that pretty much the rest of the world (in my world, windows doesn't
> exist :-) uses.
>
> > All of which is to say, the Python libraries are all "correct" for the
> > machine that compiled it. And of course we can run commands from the
> > built-in Python console. It just won't work when calling from different
> > Python from a machine that the QGIS app has been copied to.
>
> There are two related things here.
>
> 1) In general, when a program is compiled and linked, the binary is
> passed LDFLAGS that specify
>   - which libraries are needed
>   - which directories they are in (now)
>   - which directories should be added to RPATH for searching at runtime
>
> This looks like (on my NetBSD system with pkgsrc for python 3.13)
>
>   compile:      -I/usr/pkg/include/python3.13
>   link:         -Wl,-R/usr/pkg/lib -L/usr/pkg/lib -lpython3.13
>
> There are many other things that use the prefix something is built for,
> and the prefix where dependencies are, and that ends up in the built
> files.
>
> For reasons I don't understand, many people believe that they should be
> able to take programs built for one prefix and instead place them in
> some other prefix, and everything should work.  That requires a lot, and
> it usually isn't true, because most of the people that organize software
> builds think that build trees should not be randomly reparented.  Or it
> sometimes isn't true because some people think that.   Basically, you're
> at bast on thin ice or more likely fallen through.
>
> Thus about prefixes and moving things my advice is "Don't do that.  Not
> at all."
>
> 2) When building on one machine and moving the bits and running on
> another, you have to put it in the same prefix (see #1!).  And,
> everything that is present and relied on in the build machine has to be
> present in the operational machine.  Packaging systems deal with this.
>
>
> So if you are building something against QGIS on A, and want to copy
> those bits to B, you need the exact same QGIS install on B.  Read that
> as "same OS version and cpu type and installed from the same installer
> file".
>
> > I want to run them from an external script. So I try that using the
> > path to the python3.12 that lives inside the QGIS app. That gives me:
> >   ERROR 1: PROJ: proj_identify: Cannot find proj.db
> > which appears to be a conda error. So perhaps the build process for
> > official binaries uses conda? (I am aware of conda but have never used
> it).
>
> I don't know anything about conda.  I would expect that proj is part of
> the qgis build, and that the path to proj.db is built in at build time.
> This is one of those things that will be trouble if you have moved it.
> Use "find" to find proj.db, and maybe use "strings" on the proj binary
> and libs to see where it is looking, or ktrace/strace/ktruss/dtrace/?
> however it is spelled on macOS.
>
> The QGIS build could also be setting environment variables.
>
> Using the python binary inside QGIS sounds right to me.
>
> > ... what if we use python -m venv to create a copy of the correct
> > environment?
> > /Applications/QGIS-final-4_2_2.app/Contents/MacOS/python -m venv
> /tmp/venv2
> > Error: Command '['/tmp/venv2/bin/python3.12', '-m', 'ensurepip',
> > '--upgrade', '--default-pip']' returned non-zero exit status 1.
> >
> > Okay, it was worth a try.
>
> I don't think that will help, even if you get it to work.
>
> > Having done all of that, I am back to my central requirements:
> > I want to write external Python scripts / programs on my Mac in an IDE of
> > my choice  that I can do significant work with and which can leverage
> > various Python libraries not already included with QGIS.
> >
> > Based on all the above, do you all think it is more fruitful for me to
> > continue to try to build QGIS 4.2.x from source, to attempt to work with
> > the python already inside the app, or pursue some other avenue?
>
> I suspect you are close, and would recheck the whole reparenting.
>
> I'm on shaky ground here, but you should be able to use the included
> python to build a venv somehow (maybe having to build virtualenv with
> setup.py) to a different prefix, and then run the qgis python with
> PYTHONPATH set to also load from your venv.  At least that's what I
> would try.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/qgis-user/attachments/20261006/93f131ae/attachment.htm>