[QGIS-Developer] [saga-gis-developer] [GRASS-dev] External providers in QGIS

[Paolo Cavallini]

Thanks Vaclav for your detailed analysis. As expected, it seems that
there are a number of common tasks between GRASS and OTB, possibly also
SAGA, so it make sense to keep a joint discussion. Please consider
participating even remotely to the HackFest discussion:
https://github.com/qgis/QGIS/wiki/DeveloperMeetingMadeira2018#future-of-processing-providers
All the best.

Il 09/02/2018 21:33, Vaclav Petras ha scritto:
> Dear Nyall and all,
> 
> here are some of my thoughts mostly relating to GRASS GIS (although it
> may not express views of the whole GRASS GIS developer team).
> 
> On Thu, Feb 8, 2018 at 8:55 PM, Nyall Dawson <nyall.dawson at gmail.com
> <mailto:nyall.dawson at gmail.com>> wrote:
>>
>> Here's the situation as I see it:
>>
>> ------------
>> The past
>> -----------
>>
>> We (the QGIS project) have struggled to keep a bunch of providers
>> stable and available with the base QGIS install, and the current
>> maintainers (Alex and Victor, others) have (understandably) struggled
>> with the burden of maintaining these alone and the time commitment
>> required to do so. Users have been frustrated by breakage which occurs
>> when the algorithms they depend on break due to changes in underlying
>> libraries for which the processing providers have not been adapted.
>>
>> Despite this, I'd say overall it's worked OK-ish up to now, but
>> definitely with lots of room for improvement.
>>
>> -------------
>> The goals
>> -------------
>>
>> I think everyone involved is in agreement that processing's strength
>> comes when there's many providers available and it's able to tie
>> together processes regardless of which provider or library they come
>> from. So I'd say our common goals are:
>>
>> 1. Encourage development of as many processing providers as possible
>> 2. Make it easy for developers to create and maintain these providers
>> 3. Make it easy for users to be able to use the providers
>> 4. Make everything stable and painfree for users
>>
>> Any disagreements? No? Good ;)
>>
>> So if we agree that those are the end goals, we should be using them
>> to drive this discussion.
>>
>> -------------------
>> The questions
>> -------------------
>>
>> The open, debatable questions are:
>>
>> - Which is the best approach for allowing easy maintenance of
>> providers (*regardless* of whether the provider is maintained by the
>> core QGIS team or a 3rd party)? Is it keeping the code in the master
>> codebase and locking to QGIS releases, or publishing via plugins and
>> having independent release schedules? Which approach will encourage
>> more active maintenance of these providers and share the burden of
>> this maintenance?
> 
> One thing is where to keep the code, however I'm not sure what code are
> we talking about and I would like to talk about this first. I think that
> recent post by Moritz is bringing this up as well:
> 
> https://lists.osgeo.org/pipermail/grass-dev/2018-February/087420.html
> <https://lists.osgeo.org/pipermail/grass-dev/2018-February/087420.html>
> 
> As far as I know, QGIS Processing has a text file for each GRASS module
> describing its interface and maintenance of these takes time. However,
> every GRASS module can tell about itself when called with
> --interface-description parameter. If this is used, individual
> parameters don't need to be maintained and any, even user provided
> module can be executed in processing.
> 
> The --interface-description parameter gives XML which would need to be
> converted or a new parameter, let's say --qgis-description, would need
> to be added for a QGIS-specific format, there is for example --script
> for GRASS GIS interface description for scripts. --qgis-description
> would need to be in GRASS GIS code base while the conversion can be
> anywhere. See emails from Rashad discussing a possible implementation
> with the --qgis-description way and the OTB way:
> 
> https://lists.osgeo.org/pipermail/grass-dev/2018-February/087362.html
> <https://lists.osgeo.org/pipermail/grass-dev/2018-February/087362.html>
> https://lists.osgeo.org/pipermail/grass-dev/2018-February/087390.html
> <https://lists.osgeo.org/pipermail/grass-dev/2018-February/087390.html>
> 
> The was discussed in the past and in fact, it is used in the QGIS GRASS
> plugin as Radim explains in this older email:
> 
> https://lists.osgeo.org/pipermail/grass-dev/2015-March/074629.html
> <https://lists.osgeo.org/pipermail/grass-dev/2015-March/074629.html>
> 
> However, it is not using the --interface-description XML only, but it is
> also using the qgm files to supply some additional information which
> means that this system still requires updates which are separate from
> the updates of GRASS modules. This can be avoided if the necessary
> information is updated upstream or sometimes even GRASS
> --interface-description format or the individual module descriptions are
> extended to include that what is needed by QGIS Processing. Here is a
> thread which disuses this issues although diverges into the following:
> 
> https://lists.osgeo.org/pipermail/grass-dev/2015-September/076138.html
> <https://lists.osgeo.org/pipermail/grass-dev/2015-September/076138.html>
> 
> One issue which is creating differences between QGIS Processing
> representation of the module and the GRASS one is the issue of advanced
> parameters. GRASS itself has mechanism to organize the parameters into
> groups and marks the required ones. This is of course something which is
> constantly being refined and it can be used and changed also for QGIS
> Processing as discussed in this 2015 post:
> 
> https://lists.osgeo.org/pipermail/grass-dev/2015-December/078000.html
> <https://lists.osgeo.org/pipermail/grass-dev/2015-December/078000.html>
> 
> Other issues besides the interface include list of GRASS modules usable
> and unusable in QGIS Processing, thematic tree of these modules,
> splitting modules and more. Many of these are discussed in this recent post:
> 
> https://lists.osgeo.org/pipermail/grass-dev/2018-February/087388.html
> <https://lists.osgeo.org/pipermail/grass-dev/2018-February/087388.html>
> 
> One of the issues is issue of formats (or import & export). Here again,
> it seems to that it is more advantageous to have a function in GRASS
> code base to care take of the import and, if needed, push the changes
> upstream to it (from QGIS point of view). Here is a related example
> which is a comment from Markus Neteler (regarding currently preferred
> solution for vector import) which would be part of PR/patch review
> process if the import function was originally submitted as extension of
> GRASS code base:
> 
> https://github.com/qgis/QGIS/pull/5426#issuecomment-345067457
> <https://github.com/qgis/QGIS/pull/5426#issuecomment-345067457>
> 
> Then there is of course the issue of GRASS modules which are more
> difficult to wrap in QGIS Processing because the inputs and outputs are
> not explicitly stated, most prominent example is probably r.mapcalc but
> there is couple of more, although we were able to reduce this number lately:
> 
> https://lists.osgeo.org/pipermail/grass-dev/2015-January/072979.html
> <https://lists.osgeo.org/pipermail/grass-dev/2015-January/072979.html>
> 
> See also the following post from Giovanni Manghi (sent off-list
> to Markus Neteler and shared by him) and a ticket commend by me for more
> examples and details:
> 
> https://lists.osgeo.org/pipermail/grass-dev/2014-August/070342.html
> <https://lists.osgeo.org/pipermail/grass-dev/2014-August/070342.html>
> https://trac.osgeo.org/grass/ticket/2409#comment:17
> <https://trac.osgeo.org/grass/ticket/2409#comment:17>
> 
> Here is a good r.mapcalc-related example of (at least) asking for
> changes upstream (although sent privately to Markus Neteler who opened
> the issue):
> 
> https://trac.osgeo.org/grass/ticket/3431
> <https://trac.osgeo.org/grass/ticket/3431>
> 
>> - Who should be responsible for maintaining the providers? Should
>> maintenance be done by the QGIS core developer team or by 3rd parties?
> 
> I'm not sure if we can really separate this from the place in the code,
> but if there is a notion of pushing changes upstream then that gives
> definitively us more flexibility.
> 
> Besides maintenance, I think the initial development, which is necessary
> for a realistic maintenance plan, is a major issue. I, and I think some
> other GRASS devs too, would be happy to work on it if the financing is
> addressed. A different suggestion is here as well and that's GSoC idea
> proposed by Martin Landa:
> 
> https://trac.osgeo.org/grass/wiki/GSoC/2018#ImproveGRASSintegrationinQGIS3
> <https://trac.osgeo.org/grass/wiki/GSoC/2018#ImproveGRASSintegrationinQGIS3>
> 
>> - Is it a core requirement that 3rd party providers are installed with
>> EVERY QGIS install, or is it acceptable to require interested users to
>> manually install the tools which they find useful? If the second
>> option is preferred, then how can we ensure that users are aware of
>> the availability of these tools?
> 
> If they install manually, on Windows, users can have the latest version
> of the software instead of the one packaged with QGIS. OSGeo4W is a
> different case; not sure about Mac. However, on Linux, the decision is
> up to the packagers, so in any case, the QGIS part needs to be able to
> work with different versions. Regardless of QGIS packaging efforts, I
> have seen many users with old GRASS who installed new standalone GRASS
> on the side, even recently.
> 
> On the other hand, many users benefit from the one package, because they
> need to get software approved, if they get QGIS packaged with GRASS
> approve, they got GRASS as well. Getting approval for each individual
> provider is likely more paperwork.
> 
> I hope this will help and let me know what I'm missing.
> 
> Best,
> Vaclav
> 
> 
> ------------------------------------------------------------------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> 
> 
> 
> _______________________________________________
> saga-gis-developer mailing list
> saga-gis-developer at lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/saga-gis-developer
> 


-- 
Paolo Cavallini - www.faunalia.eu
QGIS & PostGIS courses: http://www.faunalia.eu/training.html
https://www.google.com/trends/explore?date=all&geo=IT&q=qgis,arcgis