[QGIS-Developer] Understanding plugin management in QGIS - exposing QgsPluginRegistry to Python?

[Sebastian M. Ernst]

Hi Nyall, Paolo, Alexander, Javier,

> The issues we see most often stem not from any particular use of
> Python, but (as Matthias has pointed out) through the nature of
> Pythons weak typing and lack of compile time checks. A simple example:
> if I alter the signature of a virtual method in one of the c++ classes
> to add an extra argument or change a return type, I'll know right away
> (well, when I next compile) if I've broken some existing code. If I do
> the same in Python, I'm basically trusting that my particular IDE can
> correctly identify ALL use of that method, because I won't get any
> warning if I did break something. Unit tests help, but will never
> completely avoid this kind of breakage.

agreed. One can significantly reduce the risk with the right tooling, a
lot of discipline, experience and tests. But one can never rule this
sort of thing out entirely. It is the nature of what Python allows you
to do (whether or not some of this stuff can be considered good or bad
practice) vs. what static analysis tools can understand. Most of what I
am interested in doing here does not require pitch-black Python magic,
so a good tool-chain should be able to handle it if the code is written
adequately (at least in my personal experience).

Javier, you alluded on type hints. Yes, they are one of the interesting
options one has in modern Python. And if your code is not too
performance-critical, there are actually ways to make the interpreter
respect then at run-time, i.e. Python with static types in
function/method signatures :) None of this is currently being used in QGIS.

> Another HUGE factor with Python+QGIS is that the actual methods of
> creating the Python bindings + the PyQt library itself is a mysterious
> black box, full of weird inconsistencies, random results, memory
> mismanagement and crashy code practices. I shudder to think of the
> amount of time I've wasted fighting with SIP and its unpredictable
> behavior. You could write the BEST Python code the world has ever
> seen, using all the Python best practice coding techniques you want,
> and you'll still hit random regressions and issues due to these
> libraries. QGIS c++ code is just MUCH more predictable, transparent,
> and easier to debug.

Ok, this sounds like serious issues with SIP. The smaller your SIP
interface is, the smaller your surface of attack becomes. Which makes me
wonder ... even if you shift as much of QGIS' logic from Python to C++,
you'll always be stuck with some SIP due to your Python integration and
the plugin ecosystem. From what I have seen, QGIS (in its current form)
is the ultimate stress-test for SIP as in QGIS does some really wild
stuff with it. Would not it make sense to clean up a lot of those SIP
interfaces and limit them to what actually makes sense?

> But please please please can I ask you to seriously reconsider your
> approach? I've been working on this project for nearly 10 years now,
> and my experience time and time again is that large blocks of Python
> code in QGIS end up unstable. Modern c++ + qt is really not that much
> trickier to write then Python, and if you include good test coverage,
> then the code becomes almost self-maintaining*.

I have not seen any self-maintaining code in my life ;)

Personally, but that's just a non-technical preference, I am way faster
& better with prototyping / developing in Python than I am in C++. I
usually port code away from Python in a later work step once stuff is
working (and tested). Other than that, while understanding your points,
I can still see how a Python solution / prototype could make sense from
a technical perspective and still address your concerns. Allow me to
elaborate on it.

I want to manage Python plugins, Python packages and Python dependencies
as well as load, unload and inspect Python code while remaining on top
of things by catching and treating all possible exceptions in this
process. Therefore, I want to work closely with some Python modules,
e.g. `ast`, `inspect`, `importlib` and `conda.api`. I am heavily tapping
into the interpreter for that. The closer I remain to my objective, the
more flexibility I have and the more stability I gain. Even if I move as
much as possible over to C++, i.e. the GUI and some logic, the actually
required Python layer (plus C++ interface code) would remain
substantial. It would also lead to a convoluted logic, probably in the
end only little better than the current plugin installer. This is
precisely what I want to get away from. Having said that, the plugin
manager is - from my perspective at least - different from processing or
the db manager. They are significantly different technical problems.
While I agree with you that, due to SIP, it is a good idea to move as
much of processing and db manager over to C++, I am thinking that going
the opposite direction specifically *here* makes more sense. While QGIS
is a hybrid, I'd attempt to limit my exposure to SIP by limiting my
interaction with QGIS (and PyQt) APIs to a bare minimum. After all, I
see a good plugin manager more as an "independent" Python package (with
all of its typical infrastructure) than something that is heavily woven
into QGIS core (while it might actually be distributed / deployed with
QGIS core).

How does this sound?

Best regards,
Sebastian