[GRASS-dev] [QGIS-Developer] External providers in QGIS

[Vaclav Petras]

Dear Nyall and all,

here are some of my thoughts mostly relating to GRASS GIS (although it may
not express views of the whole GRASS GIS developer team).

On Thu, Feb 8, 2018 at 8:55 PM, Nyall Dawson <nyall.dawson at gmail.com> wrote:
>
> Here's the situation as I see it:
>
> ------------
> The past
> -----------
>
> We (the QGIS project) have struggled to keep a bunch of providers
> stable and available with the base QGIS install, and the current
> maintainers (Alex and Victor, others) have (understandably) struggled
> with the burden of maintaining these alone and the time commitment
> required to do so. Users have been frustrated by breakage which occurs
> when the algorithms they depend on break due to changes in underlying
> libraries for which the processing providers have not been adapted.
>
> Despite this, I'd say overall it's worked OK-ish up to now, but
> definitely with lots of room for improvement.
>
> -------------
> The goals
> -------------
>
> I think everyone involved is in agreement that processing's strength
> comes when there's many providers available and it's able to tie
> together processes regardless of which provider or library they come
> from. So I'd say our common goals are:
>
> 1. Encourage development of as many processing providers as possible
> 2. Make it easy for developers to create and maintain these providers
> 3. Make it easy for users to be able to use the providers
> 4. Make everything stable and painfree for users
>
> Any disagreements? No? Good ;)
>
> So if we agree that those are the end goals, we should be using them
> to drive this discussion.
>
> -------------------
> The questions
> -------------------
>
> The open, debatable questions are:
>
> - Which is the best approach for allowing easy maintenance of
> providers (*regardless* of whether the provider is maintained by the
> core QGIS team or a 3rd party)? Is it keeping the code in the master
> codebase and locking to QGIS releases, or publishing via plugins and
> having independent release schedules? Which approach will encourage
> more active maintenance of these providers and share the burden of
> this maintenance?

One thing is where to keep the code, however I'm not sure what code are we
talking about and I would like to talk about this first. I think that
recent post by Moritz is bringing this up as well:

https://lists.osgeo.org/pipermail/grass-dev/2018-February/087420.html

As far as I know, QGIS Processing has a text file for each GRASS module
describing its interface and maintenance of these takes time. However,
every GRASS module can tell about itself when called with
--interface-description parameter. If this is used, individual parameters
don't need to be maintained and any, even user provided module can be
executed in processing.

The --interface-description parameter gives XML which would need to be
converted or a new parameter, let's say --qgis-description, would need to
be added for a QGIS-specific format, there is for example --script for
GRASS GIS interface description for scripts. --qgis-description would need
to be in GRASS GIS code base while the conversion can be anywhere. See
emails from Rashad discussing a possible implementation with the
--qgis-description way and the OTB way:

https://lists.osgeo.org/pipermail/grass-dev/2018-February/087362.html
https://lists.osgeo.org/pipermail/grass-dev/2018-February/087390.html

The was discussed in the past and in fact, it is used in the QGIS GRASS
plugin as Radim explains in this older email:

https://lists.osgeo.org/pipermail/grass-dev/2015-March/074629.html

However, it is not using the --interface-description XML only, but it is
also using the qgm files to supply some additional information which means
that this system still requires updates which are separate from the updates
of GRASS modules. This can be avoided if the necessary information is
updated upstream or sometimes even GRASS --interface-description format or
the individual module descriptions are extended to include that what is
needed by QGIS Processing. Here is a thread which disuses this issues
although diverges into the following:

https://lists.osgeo.org/pipermail/grass-dev/2015-September/076138.html

One issue which is creating differences between QGIS Processing
representation of the module and the GRASS one is the issue of advanced
parameters. GRASS itself has mechanism to organize the parameters into
groups and marks the required ones. This is of course something which is
constantly being refined and it can be used and changed also for QGIS
Processing as discussed in this 2015 post:

https://lists.osgeo.org/pipermail/grass-dev/2015-December/078000.html

Other issues besides the interface include list of GRASS modules usable and
unusable in QGIS Processing, thematic tree of these modules, splitting
modules and more. Many of these are discussed in this recent post:

https://lists.osgeo.org/pipermail/grass-dev/2018-February/087388.html

One of the issues is issue of formats (or import & export). Here again, it
seems to that it is more advantageous to have a function in GRASS code base
to care take of the import and, if needed, push the changes upstream to it
(from QGIS point of view). Here is a related example which is a comment
from Markus Neteler (regarding currently preferred solution for vector
import) which would be part of PR/patch review process if the import
function was originally submitted as extension of GRASS code base:

https://github.com/qgis/QGIS/pull/5426#issuecomment-345067457

Then there is of course the issue of GRASS modules which are more difficult
to wrap in QGIS Processing because the inputs and outputs are not
explicitly stated, most prominent example is probably r.mapcalc but there
is couple of more, although we were able to reduce this number lately:

https://lists.osgeo.org/pipermail/grass-dev/2015-January/072979.html

See also the following post from Giovanni Manghi (sent off-list to Markus
Neteler and shared by him) and a ticket commend by me for more examples and
details:

https://lists.osgeo.org/pipermail/grass-dev/2014-August/070342.html
https://trac.osgeo.org/grass/ticket/2409#comment:17

Here is a good r.mapcalc-related example of (at least) asking for changes
upstream (although sent privately to Markus Neteler who opened the issue):

https://trac.osgeo.org/grass/ticket/3431

> - Who should be responsible for maintaining the providers? Should
> maintenance be done by the QGIS core developer team or by 3rd parties?

I'm not sure if we can really separate this from the place in the code, but
if there is a notion of pushing changes upstream then that gives
definitively us more flexibility.

Besides maintenance, I think the initial development, which is necessary
for a realistic maintenance plan, is a major issue. I, and I think some
other GRASS devs too, would be happy to work on it if the financing is
addressed. A different suggestion is here as well and that's GSoC idea
proposed by Martin Landa:

https://trac.osgeo.org/grass/wiki/GSoC/2018#ImproveGRASSintegrationinQGIS3

> - Is it a core requirement that 3rd party providers are installed with
> EVERY QGIS install, or is it acceptable to require interested users to
> manually install the tools which they find useful? If the second
> option is preferred, then how can we ensure that users are aware of
> the availability of these tools?

If they install manually, on Windows, users can have the latest version of
the software instead of the one packaged with QGIS. OSGeo4W is a different
case; not sure about Mac. However, on Linux, the decision is up to the
packagers, so in any case, the QGIS part needs to be able to work with
different versions. Regardless of QGIS packaging efforts, I have seen many
users with old GRASS who installed new standalone GRASS on the side, even
recently.

On the other hand, many users benefit from the one package, because they
need to get software approved, if they get QGIS packaged with GRASS
approve, they got GRASS as well. Getting approval for each individual
provider is likely more paperwork.

I hope this will help and let me know what I'm missing.

Best,
Vaclav
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/grass-dev/attachments/20180209/f4716358/attachment.html>