[QGIS-Developer] Understanding plugin management in QGIS - exposing QgsPluginRegistry to Python?

[Nyall Dawson]

On Sun, 15 Mar 2020 at 21:50, Sebastian M. Ernst <ernst at pleiszenburg.de> wrote:
>
> Hi Borys, Nyall, Matthias,
>
> thanks all for your replies! Let me address some of the points/concerns
> raised.
>
> > Actually, over time we've found that the opposite is true -- the
> > Python code in QGIS master is the **most** work to maintain, and a
> > constant source of regressions. That's due to the nature of the Python
> > language itself, not a reflection of the quality of the code...
>
> Nyall, could you point me to some specific regressions that you are
> referring to? Bugs or their IDs (so I can look them up)? When you say
> "nature of the Python language", which specific aspects are you
> referring to? I know there is a lot that one can legitimately criticize
> about the language, but again I'd be highly interested in what
> specifically is causing you headaches.

The issues we see most often stem not from any particular use of
Python, but (as Matthias has pointed out) through the nature of
Pythons weak typing and lack of compile time checks. A simple example:
if I alter the signature of a virtual method in one of the c++ classes
to add an extra argument or change a return type, I'll know right away
(well, when I next compile) if I've broken some existing code. If I do
the same in Python, I'm basically trusting that my particular IDE can
correctly identify ALL use of that method, because I won't get any
warning if I did break something. Unit tests help, but will never
completely avoid this kind of breakage.

And we see this happen ALL. THE. TIME. in QGIS Python code. Processing
(through NO fault of the maintainers) was a frequent source of
breakage in QGIS 2.x, and it was only by porting the guts of it away
from Python and to c++ that we could finally get things stable. Now
it's happening all the time in dbmanager -- I don't think there's been
a single recent QGIS release which hasn't had some form of regression
in db manager. And again, I blame that on the nature of Python, not on
the existing code or maintainers.

Another HUGE factor with Python+QGIS is that the actual methods of
creating the Python bindings + the PyQt library itself is a mysterious
black box, full of weird inconsistencies, random results, memory
mismanagement and crashy code practices. I shudder to think of the
amount of time I've wasted fighting with SIP and its unpredictable
behavior. You could write the BEST Python code the world has ever
seen, using all the Python best practice coding techniques you want,
and you'll still hit random regressions and issues due to these
libraries. QGIS c++ code is just MUCH more predictable, transparent,
and easier to debug.

>> That's due to the nature of the Python
> > language itself, not a reflection of the quality of the code...
> Reading the code of the Python (environment) integration, I can see that
> it is prone to all sorts of issues - but I'd be interested in your
> experience in what actually causes problems the most.

My point wasn't a comment of the quality of the Python code either
way. Rather, my point was that **regardless** of the quality of the
Python code, you'll still end up with a difficult to maintain, fragile
result!

> > That would be my preference too -- the overall movement in QGIS code
> > is to reduce the overall amount of Python code in master, not the
> > reverse.
>
> I can understand where you are coming from, however please allow me to
> suggest the following: I would like to do what I already hinted on, i.e.
> unify most plugin management code into a common, modern Python code base
> with tests etc.

This work sounds fantastic, and highly valuable to QGIS. It's GREAT to
hear that you want to investigate this!!

But please please please can I ask you to seriously reconsider your
approach? I've been working on this project for nearly 10 years now,
and my experience time and time again is that large blocks of Python
code in QGIS end up unstable. Modern c++ + qt is really not that much
trickier to write then Python, and if you include good test coverage,
then the code becomes almost self-maintaining*.

Python is a GREAT choice for QGIS scripts and addons, but I just don't
think it's a suitable choice here.

Nyall

* at least, until a major qt release deprecates stuff you rely on! :P



I would like to show how this could look like. You do
> not have to accept it or merge it. After all, I would probably have to
> write an enhancement proposal first. So, please take it as a proof of
> concept for now. How about that?
>
> Once I have a working proof of concept, I actually have a few ideas on
> how to take it forward and add a few interesting features to it - again
> as proof of concept. One key idea is the integration of the conda
> package manager. Other than pip, it actually has an official, stable
> API. And, well, QGIS has recently been packaged for Anaconda for all
> relevant platforms (kudos and thanks to everyone involved!). A conda
> integration has a hell of a lot of potential.
>
> > I admit the the infrastructure became a bit complicated ;) However, porting
> > the Plugin Manager to Python would require making Python a hard dependency of
> > QGIS GUI (or we should drop C++ plugins by porting them all to core, what is
> > discussed from time to time and doesn't seem to may happen).
>
> Ok, so Python has to remain optional. This is just fine. As far as I can
> see it, C++ plugins have to be compiled with QGIS anyway. At runtime,
> all a user can do is to activate or deactivate them. Borys, am I
> understanding this correctly? If a *minimal* C++ GUI would allow exactly
> that, you could reduce a lot of the complexity. The rest could
> (following my instinct) cleanly be written in Python. I.e. if (and only
> if) Python is present, a fully featured plugin manager (for C++ *and*
> Python plugins) could drop in and *replace* the simple C++ plugin on/off
> manager GUI. How about that?
>
> If you're going in the opposite direction, doing all in C++, you're
> getting stuck with yet another hybrid solution, i.e. calling from C++
> into Python (not only the other way round). No matter how thin the
> remaining Python layer is, some of your current problems are likely not
> going to go away. The beauty of going (almost full) Python is that
> Python plugins can be managed by a Python manager - no hybrid required.
> If done properly, as I said before, only very few calls from Python into
> QGIS' C++ API would remain - essentially for activating and deactivating
> the C++ plugins. The manager could then, theoretically, even be made to
> optionally run stand-alone without QGIS, which leads to significantly
> easier testing with a lot less mocking.
>
> If I want to develop my proof of concept (and actually proof that this
> could work), I'd ideally need access to at least some methods from
> `QgsPluginRegistry`. Back to my original question: Can this class be
> accessed from Python? Or could someone help me with exposing the
> relevant methods (from one/the relevant instance of this class) to
> Python? I have no trouble with building QGIS from source if required.
>
> Best regards,
> Sebastian
> _______________________________________________
> QGIS-Developer mailing list
> QGIS-Developer at lists.osgeo.org
> List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer