[Qgis-user] [Caution - External w/link] Re: Troubleshooting Python Script - QGIS library Import on Recent Versions on Mac OS X

[Greg Troxel]

"Odom, Johnnie" <JOdom at ecsdfl.us> writes:

> Thank you for all of your advice and help. I've been intermittently trying
> to troubleshoot ever since. I'm writing for anyone who comes across this
> thread later, and to throw out a few additional possibilities for group
> discussion.

There's a lot in there and I think this digging in and documenting is
helpful.

> On the Mac, the equivalent of ldd is "otool -L".
> Without explaining the rabbit holes I have been down the last several
> weeks, I can simply say that the dylib situation on OS X is complicated.

Agreed; dylib is more complicated and less understood than ELF libraries
that pretty much the rest of the world (in my world, windows doesn't
exist :-) uses.

> All of which is to say, the Python libraries are all "correct" for the
> machine that compiled it. And of course we can run commands from the
> built-in Python console. It just won't work when calling from different
> Python from a machine that the QGIS app has been copied to.

There are two related things here.

1) In general, when a program is compiled and linked, the binary is
passed LDFLAGS that specify
  - which libraries are needed
  - which directories they are in (now)
  - which directories should be added to RPATH for searching at runtime

This looks like (on my NetBSD system with pkgsrc for python 3.13)

  compile:	-I/usr/pkg/include/python3.13
  link:		-Wl,-R/usr/pkg/lib -L/usr/pkg/lib -lpython3.13

There are many other things that use the prefix something is built for,
and the prefix where dependencies are, and that ends up in the built
files.

For reasons I don't understand, many people believe that they should be
able to take programs built for one prefix and instead place them in
some other prefix, and everything should work.  That requires a lot, and
it usually isn't true, because most of the people that organize software
builds think that build trees should not be randomly reparented.  Or it
sometimes isn't true because some people think that.   Basically, you're
at bast on thin ice or more likely fallen through.

Thus about prefixes and moving things my advice is "Don't do that.  Not
at all."

2) When building on one machine and moving the bits and running on
another, you have to put it in the same prefix (see #1!).  And,
everything that is present and relied on in the build machine has to be
present in the operational machine.  Packaging systems deal with this.


So if you are building something against QGIS on A, and want to copy
those bits to B, you need the exact same QGIS install on B.  Read that
as "same OS version and cpu type and installed from the same installer
file".

> I want to run them from an external script. So I try that using the
> path to the python3.12 that lives inside the QGIS app. That gives me:
>   ERROR 1: PROJ: proj_identify: Cannot find proj.db
> which appears to be a conda error. So perhaps the build process for
> official binaries uses conda? (I am aware of conda but have never used it).

I don't know anything about conda.  I would expect that proj is part of
the qgis build, and that the path to proj.db is built in at build time.
This is one of those things that will be trouble if you have moved it.
Use "find" to find proj.db, and maybe use "strings" on the proj binary
and libs to see where it is looking, or ktrace/strace/ktruss/dtrace/?
however it is spelled on macOS.

The QGIS build could also be setting environment variables.

Using the python binary inside QGIS sounds right to me.

> ... what if we use python -m venv to create a copy of the correct
> environment?
> /Applications/QGIS-final-4_2_2.app/Contents/MacOS/python -m venv /tmp/venv2
> Error: Command '['/tmp/venv2/bin/python3.12', '-m', 'ensurepip',
> '--upgrade', '--default-pip']' returned non-zero exit status 1.
>
> Okay, it was worth a try.

I don't think that will help, even if you get it to work.

> Having done all of that, I am back to my central requirements:
> I want to write external Python scripts / programs on my Mac in an IDE of
> my choice  that I can do significant work with and which can leverage
> various Python libraries not already included with QGIS.
>
> Based on all the above, do you all think it is more fruitful for me to
> continue to try to build QGIS 4.2.x from source, to attempt to work with
> the python already inside the app, or pursue some other avenue?

I suspect you are close, and would recheck the whole reparenting.

I'm on shaky ground here, but you should be able to use the included
python to build a venv somehow (maybe having to build virtualenv with
setup.py) to a different prefix, and then run the qgis python with
PYTHONPATH set to also load from your venv.  At least that's what I
would try.