[GRASS-dev] [GRASS-user] Keeping GRASS/OTB/... algorithm in qgis processing

Wed Feb 7 01:51:36 PST 2018

Hello,

On Wed, Feb 7, 2018 at 12:32 AM, Vaclav Petras <wenzeslaus at gmail.com> wrote:

> Dear all,
>
> I'm answering here rather than the "[GRASS-dev] External providers in
> QGIS" thread since here we have the technical discussion already. I'm also
> removing grass-user list.
>
> On Mon, Feb 5, 2018 at 3:56 PM, Stefan Blumentrath <
> Stefan.Blumentrath at nina.no> wrote:
>>
>> The text descriptor files have been a real pain for maintaining
>> interfaces for GRASS integration in Processing as well as the GRASS plugin.
>> In addition, they make usage of AddOns practically impossible for most of
>> the QGIS users.
>>
>
>
> There were already suggestions to change it in the past to something based
> on "--interface-description", but it always hit various issues like those
> things you mention below. However, I think that it is exactly what needs to
> be done in any case because only that is sustainable and potentially also
> reusable by other tools (besides QGIS). The reuse is quite interesting also
> in the relation to the automatic import, export and temporary location (in
> context of --exec or grass_session).
>
>
>
>>
>>
>> However, for QGIS (and this is true for both integrations), the module UI
>> was deliberately simplified by hiding / removing “advanced” option or
>> splitting modules into “sub-types”.
>>
>
> Here the question is how important it is to do (keep) this or if perhaps
> different solution would give better result. For example, splitting (in the
> QGIS interface) r.slope.aspect to r.slope and r.aspect (and more) makes it
> slower for the user to compute slope and aspect - 2x filling the form and
> 2x loading input data. Reorganization of r.slope.aspect interface may
> result in more convenient form and shorter computation than the "r.slope +
> r.aspect" solution.
>

OTB also suffers from this issue in QGIS. We are solving it with new
updates to processing provider. Maybe grass could reuse it!. Currently, I
have two blocking issues.
1) QGIS batch processing that demands a table with each row representing a
single execution of algorithm.
2) https://github.com/qgis/QGIS/pull/6272

standard processing dialog and qgis graphical modeler works after pull
request (pending merge)
For OTB, my initial thought is to disable batch processing and later work
on this part. It is important to have two of them working than none of them.
QGIS's proposition earlier was to split application which I don't think is
something right to do here.
Since grass also has similar issues, I prefer QGIS would accept fixes.

>
>
>> In addition not all modules could be meaningful added to the two QGIS
>> integrations (e.g. temporal modules in Processing).
>>
>
>
> I think there needs to be a white list or back list. It seems analogous to
> the "toolboxes" for wxGUI. All r3.* and t.* modules are out for QGIS, but I
> think the rest needs to be hand picked. The limitation of the hand picking
> is that it does not work for addons and custom modules in general.
>
> If there is an executable or algorithm found in a gis provider, it can ask
what should I do? And the provider can answer run some program with so and
so argument(s).
In case of otb, there will be a dedicated executable to deal with. IIUC
grass can say, if you have an algorithm in a directory (listed in provider,
could be relative to install prefix of GRASS),
run <module_name> --qgis-descriptor. With this, a user installing an addon
will have to do nothing to make it work with qgis.
ofcourse, once qgis found a descriptor file for an algorithm, it won't
bother ask provider what to do again or next run. Because it can read this
file and run algorithm.
The above "template" of identifying a QGIS provider algorithm can be
generic and reused in different providers.

If there is question of "trusting" on running apps in a given directory,
that is something debatable.

>
>> And finally, some require extra work on the QGIS side (like r.mapcalc).
>>
>
> We should address this ASAP in the process. Besides r.mapcalc, it is esp.
> modules which output list of maps which are either time series (e.g.
> r.sim.water with -t) or multiple results (e.g. r.texture method=asm,var
> output=basename). (There is a discussion about this somewhere on the list.)
> I'm not sure what is the current solution for i.* which take a group,
> perhaps a list or a multiband file is good (?). In relation that, t.* and
> r3.* modules could take and output a list of maps (multiband) as well.
>
>
>>
>>
>> So, I would assume that a --qgis-descriptor solution would be most
>> appropriate.
>>
>
> It can be "m.odule.name --qgis-descriptor" or "g.get.interface module=
> m.odule.name format=qgis" which can take the general (or generalized)
> --interface-description and transform it to what the processing adapter
> tool needs (or it can be just part of the adapter).
>
>
>
>> But that would still require additional work (beyond implementing a
>> parser solution) if the principles for the GRASS module UI in QGIS should
>> stay as it is, like:
>>
>> -          Tagging options and flags as advanced or basic/main/common
>>
>
> We already have the sections and also required parameters marked (in GUI),
> so the question is why it is not enough. Perhaps just fixing that is enough
> (for example description for each group). But of course if we say that the
> requirement for QGIS alg is that there is less than 5 options and no flags,
> then we need some additional system.
>
>
>> (could be an opportunity to consolidate terminology in the module UI in
>> GRASS as well)
>>
>
> I think one of the challenges brought up in the past was GRASS terminology
> versus QGIS terminology. We don't have a comparison table for that, but we
> have one for ArcGIS [1] and there are some obvious things like "vector map"
> versus "vector layer" and than GRASS' "layer of vector map".
>
> I can see 3 solutions: 1) ignore the differences, 2) come up with
> different names (I think we have been there already), or 3) have a special
> description field which is QGIS-friendly or OGC glossary compliant if
> needed.
>
> [1] https://grasswiki.osgeo.org/wiki/Terminology_comparison_betw
> een_ArcGIS_and_GRASS_GIS
>
>
>> -          Deciding which modules to use / exclude from QGIS (and how to
>> mark them)
>>
>
> We have the "toolbox" mechanism to populate things in wxGUI, so the files
> can be reused to create algorithm tree for QGIS. One needs to add the
> modules there explicitly and then additionally find addons, but we do that
> for wxGUI now.
>
>
>> -          …
>>
>> Maybe also Ondrejs work could be useful here :
>> https://trac.osgeo.org/grass/wiki/GSoC/2016/PyQtGUI ?
>>
>
>
> Yes, for the QGIS GRASS plugin. I'm not sure how much the plugin is part
> of this discussion. Not long ago, Radim worked on it a lot. Ondrej's work
> is aimed at what the C++ plugin is aimed at and what wxGUI is aimed at -
> i.e. it's a Qt interface to GRASS which could be used in GRASS or as QGIS
> plugin. I don't think it is useful for the processing part, but it is a
> good prototype to start from. Currently it misses a lot of features, it is
> basically limited reimplementation of gui/.../forms.py. There is also some
> raster rendering work from Soeren which is parallel to the C++/C part of
> the QGIS GRASS plugin.
>
> Best,
> Vaclav
>
>
>>
>>
>> Cheers
>>
>> Stefan
>>
>>
>>
>>
>>
>>
>>
>> *From:* grass-dev [mailto:grass-dev-bounces at lists.osgeo.org] *On Behalf
>> Of *Rashad Kanavath
>> *Sent:* mandag 5. februar 2018 17.06
>> *To:* Moritz Lennert <mlennert at club.worldonline.be>
>> *Cc:* grass-user at lists.osgeo.org; grass-dev at lists.osgeo.org; Helmut
>> Kudrnovsky <hellik at web.de>
>> *Subject:* Re: [GRASS-dev] [GRASS-user] Keeping GRASS/OTB/... algorithm
>> in qgis processing
>>
>>
>>
>>
>>
>>
>>
>> On Mon, Feb 5, 2018 at 2:49 PM, Moritz Lennert <
>> mlennert at club.worldonline.be> wrote:
>>
>> On 05/02/18 13:51, Helmut Kudrnovsky wrote:
>>
>> Von: "Moritz Lennert"
>>
>> I don't know how difficult it would be to create such algorithm
>> descriptions automagically.
>>
>>     https://github.com/qgis/QGIS/search?p=1&q=processing+grass
>> &type=&utf8=%E2%9C%93
>>
>> AFAIU there are some general python scripts to provide it in processing:
>>
>> e.g.
>> python/plugins/processing/algs/grass7/Grass7AlgorithmProvider.py
>>
>>
>> No, this is the provider itself, not a tool to create the descriptions.
>>
>>
>> and there are a lot of txt files providing the module interface
>>
>> e.g
>> python/plugins/processing/algs/grass7/description/r.out.png.txt
>> r.out.png
>> Export a GRASS raster map as a non-georeferenced PNG image
>> Raster (r.*)
>> QgsProcessingParameterRasterLayer|input|Input raster|None|False
>> QgsProcessingParameterNumber|compression|Compression level of PNG file
>> (0 = none, 1 = fastest, 9 = best)|QgsProcessingParameterNu
>> mber.Integer|6|True|0|9
>>
>> and other files
>>
>>
>> My question was whether it would be possible to create these description
>> files more or less automagically.
>>
>> I think the python scripts for more complex operations have to be created
>> manually.
>>
>> IIUC (from rapid reading of the threads on the qgis-developer list),
>> Rashad's suggestion was to keep the *AlgorithmProvider code in the QGIS
>> code base, but to possibly move the creation of the description and script
>> files to a plugin managed outside QGIS core, possibly by the respective
>> external software teams.
>>
>>
>>
>> Yes. your are right on track! the idea is external tools (processing
>> providers) manage descriptor files in a format requested by qgis
>> processing.
>>
>> I see already name-of-grass-module --interface-descriptor which gives an
>> xml for GRASS gui.
>>
>> what qgis want is a csv in a specific format. The contents of qgis
>> descriptor seems much less compared to --interface-descriptor.
>>
>> Correct me if I am wrong, --interface-descriptor is available in all
>> grass modules. So maybe a --qgis can do the work.
>>
>>
>>
>> This has some advantages.
>>
>> * GRASS developers are free to fix parameter name, parameter description,
>> list of modules(add and remove) changes without affecting qgis.
>>
>> * grass 7.5 has 10 modules and grass 7.9 can have 15 and same QGIS will
>> work.
>>
>> * QGIS will not need to maintain these files and keep updating/adding new
>> modules with their release process. whatever is generated from a grass
>> build/install have better integration with qgis
>>
>> * Finally this descriptors for qgis are generated with a makefile target
>> that allows users and packagers to include it.
>>
>> * QGIS can use multiple version of grass by changing install prefix
>> because descriptors are *always* found in a directory relative to install
>> prefix.
>>
>>
>>
>> QGIS provider will be like:
>>
>> I picked a descriptor file
>>
>> parse and make the ui,
>>
>> take input and execute whatever program the descriptor is to run.
>>
>>
>>
>> QGIS already manages parsing of parameters and running them. So for
>> providers who wish to be integrated in qgis will deal with a descriptor
>> file and things go fine.
>>
>>
>>
>> Having a single interface to launch all or most of toolboxes (willing to
>> contribute descriptor with installation) can have same way of execution. At
>> that point, QGIS should consider
>>
>> adding some generic code in provider and avoid plugins for such toolboxes.
>>
>>
>>
>>
>>
>>
>>
>> Moritz
>>
>>
>> _______________________________________________
>> grass-dev mailing list
>> grass-dev at lists.osgeo.org
>> https://lists.osgeo.org/mailman/listinfo/grass-dev
>>
>>
>>
>>
>>
>> --
>>
>> Regards,
>>    Rashad
>>
>> _______________________________________________
>> grass-user mailing list
>> grass-user at lists.osgeo.org
>> https://lists.osgeo.org/mailman/listinfo/grass-user
>>
>
>

-- 
Regards,
   Rashad
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/grass-dev/attachments/20180207/0353da18/attachment-0001.html>