[QGIS-Developer] Understanding plugin management in QGIS - exposing QgsPluginRegistry to Python?

[Matthias Kuhn]

Hi Sebastian

On 3/15/20 12:50 PM, Sebastian M. Ernst wrote:
> Hi Borys, Nyall, Matthias,
>
> thanks all for your replies! Let me address some of the points/concerns
> raised.
>
>> Actually, over time we've found that the opposite is true -- the
>> Python code in QGIS master is the **most** work to maintain, and a
>> constant source of regressions. That's due to the nature of the Python
>> language itself, not a reflection of the quality of the code...
> Nyall, could you point me to some specific regressions that you are
> referring to? Bugs or their IDs (so I can look them up)? When you say
> "nature of the Python language", which specific aspects are you
> referring to? I know there is a lot that one can legitimately criticize
> about the language, but again I'd be highly interested in what
> specifically is causing you headaches.

Have a look at all the issues on db manager for example. Many of them 
are caused by duck-typing, interfaces not being implemented (properly) 
by subclasses, pieces that have been forgotten during refactoring ...

I'm sure we will find explanations for each of them why they were there 
and how they could have been prevented. Very often the answer will be 
"unit tests missing".

While "unit tests" are also good for C++, it does by design already 
force a good amount of sanity checks onto code at compile time.


> Reading the code of the Python (environment) integration, I can see that
> it is prone to all sorts of issues - but I'd be interested in your
> experience in what actually causes problems the most.
>
> As for the code quality ... I do not want to offend anyone, seriously.
> *PLEASE* take it as constructive criticism. I have been heavily
> consulting on Python development for the past couple of years, so I have
> reviewed, debugged and refactored a lot of code. One aspect that
> immediately struck me when reading QGIS' Python code was the fact that
> there were (absolutely unnecessary) relics of an automatic "2to3"
> conversion all over the place. This tells a lot about the state of the
> code already. Beyond that, I have found a few really bizarre (and
> outright dangerous) uses of the "global" keyword. In a few cases for
> error handling, I believe ... Yes, there are (few) legitimate usecases
> for "global" in Python, but as far as my findings in QGIS go, none of
> those make any sense. This can be solved in a much cleaner way by
> exploiting name spaces and "mutable variables" (and exceptions). Having
> said that, most of the code cares very little about mutable data types.
> A few calls to "copy" methods in interesting places tell a story of
> running into issues related to mutability without caring too much about
> it. Well, I really do not want to insinuate a lack of understanding.
> Also, going into the same direction, the code heavily relies on the
> garbage collector doing a really good job. If this was performance
> critical, it would be very slow. If someone relying on the correct use
> of mutability would use some of the Python APIs in QGIS, it would sure
> as hell lead to layers of bugs. As far as style goes, it is really hard
> to see what internal and what external APIs are (i.e. which methods are
> only called from within classes/modules or from the outside). Because
> there is no public/private in Python, the typical convention is to use
> (single) leading underscores in names. It is a minor detail but one that
> leads to all sorts of issues when maintaining code in the longer run. A
> somewhat bigger style issue is the fact that there is almost no
> exception raising/handling as well as no parameter type, bounds and
> consistency checking in functions/methods. Again, as long as you do not
> touch this code (and stars are aligned in the right way), that's not an
> issue. It is only hitting you on the longer run.
>
> Ok, let's stop here. I really hope I have not trashed this code too much
> ... I just want to underline that (a lack of) code quality most
> definitely adds to your negative experience with Python in QGIS (core)
> from my perspective.

You certainly have some valid points here.

Anything that helps to improve the current situation of (python) code 
will be appreciated!

I would like it a lot to get a couple of pull requests that target such 
weaknesses.

Such pull requests are normally accepted quite quickly and much appreciated.

For your information: next to the plugin manager, there is also the 
Python console which in contrast to the Plugin manager is very unlikely 
to be replaced by C++ code. Any fixes in there will be long lasting and 
will help to strengthen the code for the upcoming years.

>
>> That would be my preference too -- the overall movement in QGIS code
>> is to reduce the overall amount of Python code in master, not the
>> reverse.
> I can understand where you are coming from, however please allow me to
> suggest the following: I would like to do what I already hinted on, i.e.
> unify most plugin management code into a common, modern Python code base
> with tests etc. I would like to show how this could look like. You do
> not have to accept it or merge it. After all, I would probably have to
> write an enhancement proposal first. So, please take it as a proof of
> concept for now. How about that?

No one is in the position to say no to this, the GPL grants you the 
rights for that :)

> Once I have a working proof of concept, I actually have a few ideas on
> how to take it forward and add a few interesting features to it - again
> as proof of concept. One key idea is the integration of the conda
> package manager. Other than pip, it actually has an official, stable
> API. And, well, QGIS has recently been packaged for Anaconda for all
> relevant platforms (kudos and thanks to everyone involved!). A conda
> integration has a hell of a lot of potential.

This would be a very nice addition indeed.

Plans in this direction have been on our wish list for this year. We 
have just had some discussions with Olivier about this this week.

This topic would deserve it's own QEP and I'm very interested in knowing 
a bit more about what you are after here.

>
> If I want to develop my proof of concept (and actually proof that this
> could work), I'd ideally need access to at least some methods from
> `QgsPluginRegistry`. Back to my original question: Can this class be
> accessed from Python? Or could someone help me with exMy suggestion would posing the
> relevant methods (from one/the relevant instance of this class) to
> Python? I have no trouble with building QGIS from source if required

No. It is in app, and the app library is not exposed to Python.


Best regards

Matthias