[QGIS-Developer] Understanding plugin management in QGIS - exposing QgsPluginRegistry to Python?

[Sebastian M. Ernst]

Hi Borys, Nyall, Matthias,

thanks all for your replies! Let me address some of the points/concerns
raised.

> Actually, over time we've found that the opposite is true -- the
> Python code in QGIS master is the **most** work to maintain, and a
> constant source of regressions. That's due to the nature of the Python
> language itself, not a reflection of the quality of the code...

Nyall, could you point me to some specific regressions that you are
referring to? Bugs or their IDs (so I can look them up)? When you say
"nature of the Python language", which specific aspects are you
referring to? I know there is a lot that one can legitimately criticize
about the language, but again I'd be highly interested in what
specifically is causing you headaches.

Reading the code of the Python (environment) integration, I can see that
it is prone to all sorts of issues - but I'd be interested in your
experience in what actually causes problems the most.

As for the code quality ... I do not want to offend anyone, seriously.
*PLEASE* take it as constructive criticism. I have been heavily
consulting on Python development for the past couple of years, so I have
reviewed, debugged and refactored a lot of code. One aspect that
immediately struck me when reading QGIS' Python code was the fact that
there were (absolutely unnecessary) relics of an automatic "2to3"
conversion all over the place. This tells a lot about the state of the
code already. Beyond that, I have found a few really bizarre (and
outright dangerous) uses of the "global" keyword. In a few cases for
error handling, I believe ... Yes, there are (few) legitimate usecases
for "global" in Python, but as far as my findings in QGIS go, none of
those make any sense. This can be solved in a much cleaner way by
exploiting name spaces and "mutable variables" (and exceptions). Having
said that, most of the code cares very little about mutable data types.
A few calls to "copy" methods in interesting places tell a story of
running into issues related to mutability without caring too much about
it. Well, I really do not want to insinuate a lack of understanding.
Also, going into the same direction, the code heavily relies on the
garbage collector doing a really good job. If this was performance
critical, it would be very slow. If someone relying on the correct use
of mutability would use some of the Python APIs in QGIS, it would sure
as hell lead to layers of bugs. As far as style goes, it is really hard
to see what internal and what external APIs are (i.e. which methods are
only called from within classes/modules or from the outside). Because
there is no public/private in Python, the typical convention is to use
(single) leading underscores in names. It is a minor detail but one that
leads to all sorts of issues when maintaining code in the longer run. A
somewhat bigger style issue is the fact that there is almost no
exception raising/handling as well as no parameter type, bounds and
consistency checking in functions/methods. Again, as long as you do not
touch this code (and stars are aligned in the right way), that's not an
issue. It is only hitting you on the longer run.

Ok, let's stop here. I really hope I have not trashed this code too much
... I just want to underline that (a lack of) code quality most
definitely adds to your negative experience with Python in QGIS (core)
from my perspective.

> That would be my preference too -- the overall movement in QGIS code
> is to reduce the overall amount of Python code in master, not the
> reverse.

I can understand where you are coming from, however please allow me to
suggest the following: I would like to do what I already hinted on, i.e.
unify most plugin management code into a common, modern Python code base
with tests etc. I would like to show how this could look like. You do
not have to accept it or merge it. After all, I would probably have to
write an enhancement proposal first. So, please take it as a proof of
concept for now. How about that?

Once I have a working proof of concept, I actually have a few ideas on
how to take it forward and add a few interesting features to it - again
as proof of concept. One key idea is the integration of the conda
package manager. Other than pip, it actually has an official, stable
API. And, well, QGIS has recently been packaged for Anaconda for all
relevant platforms (kudos and thanks to everyone involved!). A conda
integration has a hell of a lot of potential.

> I admit the the infrastructure became a bit complicated ;) However, porting 
> the Plugin Manager to Python would require making Python a hard dependency of 
> QGIS GUI (or we should drop C++ plugins by porting them all to core, what is 
> discussed from time to time and doesn't seem to may happen).

Ok, so Python has to remain optional. This is just fine. As far as I can
see it, C++ plugins have to be compiled with QGIS anyway. At runtime,
all a user can do is to activate or deactivate them. Borys, am I
understanding this correctly? If a *minimal* C++ GUI would allow exactly
that, you could reduce a lot of the complexity. The rest could
(following my instinct) cleanly be written in Python. I.e. if (and only
if) Python is present, a fully featured plugin manager (for C++ *and*
Python plugins) could drop in and *replace* the simple C++ plugin on/off
manager GUI. How about that?

If you're going in the opposite direction, doing all in C++, you're
getting stuck with yet another hybrid solution, i.e. calling from C++
into Python (not only the other way round). No matter how thin the
remaining Python layer is, some of your current problems are likely not
going to go away. The beauty of going (almost full) Python is that
Python plugins can be managed by a Python manager - no hybrid required.
If done properly, as I said before, only very few calls from Python into
QGIS' C++ API would remain - essentially for activating and deactivating
the C++ plugins. The manager could then, theoretically, even be made to
optionally run stand-alone without QGIS, which leads to significantly
easier testing with a lot less mocking.

If I want to develop my proof of concept (and actually proof that this
could work), I'd ideally need access to at least some methods from
`QgsPluginRegistry`. Back to my original question: Can this class be
accessed from Python? Or could someone help me with exposing the
relevant methods (from one/the relevant instance of this class) to
Python? I have no trouble with building QGIS from source if required.

Best regards,
Sebastian