[QGIS-Developer] Understanding plugin management in QGIS - exposing QgsPluginRegistry to Python?

Olivier Dalang olivier.dalang at gmail.com
Mon Mar 16 11:56:56 PDT 2020

Hi !

Thanks Sebastian for willing to work on this !

I'd suggest drafting a QEP already, as adding support for pip/conda
dependencies (which would be awesome !!) raises a few important questions
(for example whether we have virtualenvs or similar on all systems, how
plugins would declare their deps, how to deal with incompatible deps, or if
we even go one step further and support distribution qgis plugin as pip
packages, etc...). And there may be some other ideas around worth sharing
at an early stage (such as proper multiple plugin versions support, also
improving the plugin repository API - which by the way is also Python code,

As for the Python or C++ discussion, I don't have enough experience to help
decide, but what's sure is that either will be better than status quo in
terms of maintenance !



On Mon, 16 Mar 2020 at 12:10, Sebastian M. Ernst <ernst at pleiszenburg.de>

> Hi Nyall, Paolo, Alexander, Javier,
> > The issues we see most often stem not from any particular use of
> > Python, but (as Matthias has pointed out) through the nature of
> > Pythons weak typing and lack of compile time checks. A simple example:
> > if I alter the signature of a virtual method in one of the c++ classes
> > to add an extra argument or change a return type, I'll know right away
> > (well, when I next compile) if I've broken some existing code. If I do
> > the same in Python, I'm basically trusting that my particular IDE can
> > correctly identify ALL use of that method, because I won't get any
> > warning if I did break something. Unit tests help, but will never
> > completely avoid this kind of breakage.
> agreed. One can significantly reduce the risk with the right tooling, a
> lot of discipline, experience and tests. But one can never rule this
> sort of thing out entirely. It is the nature of what Python allows you
> to do (whether or not some of this stuff can be considered good or bad
> practice) vs. what static analysis tools can understand. Most of what I
> am interested in doing here does not require pitch-black Python magic,
> so a good tool-chain should be able to handle it if the code is written
> adequately (at least in my personal experience).
> Javier, you alluded on type hints. Yes, they are one of the interesting
> options one has in modern Python. And if your code is not too
> performance-critical, there are actually ways to make the interpreter
> respect then at run-time, i.e. Python with static types in
> function/method signatures :) None of this is currently being used in QGIS.
> > Another HUGE factor with Python+QGIS is that the actual methods of
> > creating the Python bindings + the PyQt library itself is a mysterious
> > black box, full of weird inconsistencies, random results, memory
> > mismanagement and crashy code practices. I shudder to think of the
> > amount of time I've wasted fighting with SIP and its unpredictable
> > behavior. You could write the BEST Python code the world has ever
> > seen, using all the Python best practice coding techniques you want,
> > and you'll still hit random regressions and issues due to these
> > libraries. QGIS c++ code is just MUCH more predictable, transparent,
> > and easier to debug.
> Ok, this sounds like serious issues with SIP. The smaller your SIP
> interface is, the smaller your surface of attack becomes. Which makes me
> wonder ... even if you shift as much of QGIS' logic from Python to C++,
> you'll always be stuck with some SIP due to your Python integration and
> the plugin ecosystem. From what I have seen, QGIS (in its current form)
> is the ultimate stress-test for SIP as in QGIS does some really wild
> stuff with it. Would not it make sense to clean up a lot of those SIP
> interfaces and limit them to what actually makes sense?
> > But please please please can I ask you to seriously reconsider your
> > approach? I've been working on this project for nearly 10 years now,
> > and my experience time and time again is that large blocks of Python
> > code in QGIS end up unstable. Modern c++ + qt is really not that much
> > trickier to write then Python, and if you include good test coverage,
> > then the code becomes almost self-maintaining*.
> I have not seen any self-maintaining code in my life ;)
> Personally, but that's just a non-technical preference, I am way faster
> & better with prototyping / developing in Python than I am in C++. I
> usually port code away from Python in a later work step once stuff is
> working (and tested). Other than that, while understanding your points,
> I can still see how a Python solution / prototype could make sense from
> a technical perspective and still address your concerns. Allow me to
> elaborate on it.
> I want to manage Python plugins, Python packages and Python dependencies
> as well as load, unload and inspect Python code while remaining on top
> of things by catching and treating all possible exceptions in this
> process. Therefore, I want to work closely with some Python modules,
> e.g. `ast`, `inspect`, `importlib` and `conda.api`. I am heavily tapping
> into the interpreter for that. The closer I remain to my objective, the
> more flexibility I have and the more stability I gain. Even if I move as
> much as possible over to C++, i.e. the GUI and some logic, the actually
> required Python layer (plus C++ interface code) would remain
> substantial. It would also lead to a convoluted logic, probably in the
> end only little better than the current plugin installer. This is
> precisely what I want to get away from. Having said that, the plugin
> manager is - from my perspective at least - different from processing or
> the db manager. They are significantly different technical problems.
> While I agree with you that, due to SIP, it is a good idea to move as
> much of processing and db manager over to C++, I am thinking that going
> the opposite direction specifically *here* makes more sense. While QGIS
> is a hybrid, I'd attempt to limit my exposure to SIP by limiting my
> interaction with QGIS (and PyQt) APIs to a bare minimum. After all, I
> see a good plugin manager more as an "independent" Python package (with
> all of its typical infrastructure) than something that is heavily woven
> into QGIS core (while it might actually be distributed / deployed with
> QGIS core).
> How does this sound?
> Best regards,
> Sebastian
> _______________________________________________
> QGIS-Developer mailing list
> QGIS-Developer at lists.osgeo.org
> List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/qgis-developer/attachments/20200316/79e58179/attachment.html>

More information about the QGIS-Developer mailing list