<div dir="ltr"><div>Dear all,<br><br></div><div>I'm answering here rather than the "[GRASS-dev] External providers in QGIS" thread since here we have the technical discussion already. I'm also removing grass-user list.<br></div><div><div><div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Feb 5, 2018 at 3:56 PM, Stefan Blumentrath <span dir="ltr"><<a href="mailto:Stefan.Blumentrath@nina.no" target="_blank">Stefan.Blumentrath@nina.no</a>></span> wrote:<span style="font-size:11pt;font-family:"Calibri",sans-serif;color:rgb(31,73,125)" lang="EN-GB"></span><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div lang="NO-BOK"><div class="gmail-m_-6913107978359603392m_1928144714898603219WordSection1">
<p class="MsoNormal"><span style="font-size:11pt;font-family:"Calibri",sans-serif;color:rgb(31,73,125)" lang="EN-GB">The text descriptor files have been a real pain for maintaining interfaces for GRASS integration in Processing as well
as the GRASS plugin. In addition, they make usage of AddOns practically impossible for most of the QGIS users.</span></p></div></div></blockquote><div><br><br></div><div>There were already suggestions to change it in the past to something based on "--interface-description", but it always hit various issues like those things you mention below. However, I think that it is exactly what needs to be done in any case because only that is sustainable and potentially also reusable by other tools (besides QGIS). The reuse is quite interesting also in the relation to the automatic import, export and temporary location (in context of --exec or grass_session).<br></div><div><br> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div lang="NO-BOK"><div class="gmail-m_-6913107978359603392m_1928144714898603219WordSection1"><p class="MsoNormal"><span style="font-size:11pt;font-family:"Calibri",sans-serif;color:rgb(31,73,125)" lang="EN-GB"><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:"Calibri",sans-serif;color:rgb(31,73,125)" lang="EN-GB"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:"Calibri",sans-serif;color:rgb(31,73,125)" lang="EN-GB">However, for QGIS (and this is true for both integrations), the module UI was deliberately simplified by hiding / removing
“advanced” option or splitting modules into “sub-types”.</span></p></div></div></blockquote><div><br></div><div>Here the question is how important it is to do (keep) this or if perhaps different solution would give better result. For example, splitting (in the QGIS interface) r.slope.aspect to r.slope and r.aspect (and more) makes it slower for the user to compute slope and aspect - 2x filling the form and 2x loading input data. Reorganization of r.slope.aspect interface may result in more convenient form and shorter computation than the "r.slope + r.aspect" solution.<br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div lang="NO-BOK"><div class="gmail-m_-6913107978359603392m_1928144714898603219WordSection1"><p class="MsoNormal"><span style="font-size:11pt;font-family:"Calibri",sans-serif;color:rgb(31,73,125)" lang="EN-GB"> In addition not all modules could be meaningful added to the two QGIS integrations (e.g. temporal modules in Processing).</span></p></div></div></blockquote><div><br><br></div><div>I think there needs to be a white list or back list. It seems analogous to the "toolboxes" for wxGUI. All r3.* and t.* modules are out for QGIS, but I think the rest needs to be hand picked. The limitation of the hand picking is that it does not work for addons and custom modules in general.<br></div><div><br> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div lang="NO-BOK"><div class="gmail-m_-6913107978359603392m_1928144714898603219WordSection1"><p class="MsoNormal"><span style="font-size:11pt;font-family:"Calibri",sans-serif;color:rgb(31,73,125)" lang="EN-GB"> And finally, some require extra work on the QGIS side (like r.mapcalc).</span></p></div></div></blockquote><div><br></div><div>We should address this ASAP in the process. Besides r.mapcalc, it is esp. modules which output list of maps which are either time series (e.g. r.sim.water with -t) or multiple results (e.g. r.texture method=asm,var output=basename). (There is a discussion about this somewhere on the list.) I'm not sure what is the current solution for i.* which take a group, perhaps a list or a multiband file is good (?). In relation that, t.* and r3.* modules could take and output a list of maps (multiband) as well.<br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div lang="NO-BOK"><div class="gmail-m_-6913107978359603392m_1928144714898603219WordSection1"><p class="MsoNormal"><span style="font-size:11pt;font-family:"Calibri",sans-serif;color:rgb(31,73,125)" lang="EN-GB"><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:"Calibri",sans-serif;color:rgb(31,73,125)" lang="EN-GB"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:"Calibri",sans-serif;color:rgb(31,73,125)" lang="EN-GB">So, I would assume that a --qgis-descriptor solution would be most appropriate.</span></p></div></div></blockquote><div><br></div><div>It can be "<a href="http://m.odule.name">m.odule.name</a> --qgis-descriptor" or "g.get.interface module=<a href="http://m.odule.name">m.odule.name</a> format=qgis" which can take the general (or generalized) --interface-description and transform it to what the processing adapter tool needs (or it can be just part of the adapter).<br></div><div><br> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div lang="NO-BOK"><div class="gmail-m_-6913107978359603392m_1928144714898603219WordSection1"><p class="MsoNormal"><span style="font-size:11pt;font-family:"Calibri",sans-serif;color:rgb(31,73,125)" lang="EN-GB"> But that would still require additional
work (beyond implementing a parser solution) if the principles for the GRASS module UI in QGIS should stay as it is, like:<u></u><u></u></span></p>
<p class="gmail-m_-6913107978359603392m_1928144714898603219MsoListParagraph"><u></u><span style="font-size:11pt;font-family:"Calibri",sans-serif;color:rgb(31,73,125)" lang="EN-GB"><span>-<span style="font:normal normal normal normal 7pt "Times New Roman"">
</span></span></span><u></u><span style="font-size:11pt;font-family:"Calibri",sans-serif;color:rgb(31,73,125)" lang="EN-GB">Tagging options and flags as advanced or basic/main/common</span></p></div></div></blockquote><div><br>We already have the sections and also required parameters marked (in
GUI), so the question is why it is not enough. Perhaps just fixing that
is enough (for example description for each group). But of course if we say that the requirement for QGIS alg is that there is less than 5 options and no flags, then we need some additional system.<br> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div lang="NO-BOK"><div class="gmail-m_-6913107978359603392m_1928144714898603219WordSection1"><p class="gmail-m_-6913107978359603392m_1928144714898603219MsoListParagraph"><span style="font-size:11pt;font-family:"Calibri",sans-serif;color:rgb(31,73,125)" lang="EN-GB"> (could be an opportunity to consolidate terminology
in the module UI in GRASS as well)<u></u><u></u></span></p></div></div></blockquote><div><br></div><div>I think one of the challenges brought up in the past was GRASS terminology versus QGIS terminology. We don't have a comparison table for that, but we have one for ArcGIS [1] and there are some obvious things like "vector map" versus "vector layer" and than GRASS' "layer of vector map".<br><br>I can see 3 solutions: 1) ignore the differences, 2) come up with different names (I think we have been there already), or 3) have a special description field which is QGIS-friendly or OGC glossary compliant if needed.<br><br>[1] <a href="https://grasswiki.osgeo.org/wiki/Terminology_comparison_between_ArcGIS_and_GRASS_GIS">https://grasswiki.osgeo.org/wiki/Terminology_comparison_between_ArcGIS_and_GRASS_GIS</a><br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div lang="NO-BOK"><div class="gmail-m_-6913107978359603392m_1928144714898603219WordSection1"><p class="gmail-m_-6913107978359603392m_1928144714898603219MsoListParagraph"><span style="font-size:11pt;font-family:"Calibri",sans-serif;color:rgb(31,73,125)" lang="EN-GB"></span></p>
<p class="gmail-m_-6913107978359603392m_1928144714898603219MsoListParagraph"><u></u><span style="font-size:11pt;font-family:"Calibri",sans-serif;color:rgb(31,73,125)" lang="EN-GB"><span>-<span style="font:normal normal normal normal 7pt "Times New Roman"">
</span></span></span><u></u><span style="font-size:11pt;font-family:"Calibri",sans-serif;color:rgb(31,73,125)" lang="EN-GB">Deciding which modules to use / exclude from QGIS (and how to mark them)</span></p></div></div></blockquote><div><br></div><div>We have the "toolbox" mechanism to populate things in wxGUI, so the files can be reused to create algorithm tree for QGIS. One needs to add the modules there explicitly and then additionally find addons, but we do that for wxGUI now.<br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div lang="NO-BOK"><div class="gmail-m_-6913107978359603392m_1928144714898603219WordSection1"><p class="gmail-m_-6913107978359603392m_1928144714898603219MsoListParagraph"><span style="font-size:11pt;font-family:"Calibri",sans-serif;color:rgb(31,73,125)" lang="EN-GB"><u></u><u></u></span></p>
<p class="gmail-m_-6913107978359603392m_1928144714898603219MsoListParagraph"><u></u><span style="font-size:11pt;font-family:"Calibri",sans-serif;color:rgb(31,73,125)" lang="EN-GB"><span>-<span style="font:normal normal normal normal 7pt "Times New Roman"">
</span></span></span><u></u><span style="font-size:11pt;font-family:"Calibri",sans-serif;color:rgb(31,73,125)" lang="EN-GB">…<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:"Calibri",sans-serif;color:rgb(31,73,125)" lang="EN-GB">Maybe also Ondrejs work could be useful here :
<a href="https://trac.osgeo.org/grass/wiki/GSoC/2016/PyQtGUI" target="_blank">https://trac.osgeo.org/grass/w<wbr>iki/GSoC/2016/PyQtGUI</a> ?</span></p></div></div></blockquote><div><br><br></div><div>Yes, for the QGIS GRASS plugin. I'm not sure how much the plugin is part of this discussion. Not long ago, Radim worked on it a lot. Ondrej's work is aimed at what the C++ plugin is aimed at and what wxGUI is aimed at - i.e. it's a Qt interface to GRASS which could be used in GRASS or as QGIS plugin. I don't think it is useful for the processing part, but it is a good prototype to start from. Currently it misses a lot of features, it is basically limited reimplementation of gui/.../forms.py. There is also some raster rendering work from Soeren which is parallel to the C++/C part of the QGIS GRASS plugin.<br></div><div><br></div><div>Best,<br></div><div>Vaclav<br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div lang="NO-BOK"><div class="gmail-m_-6913107978359603392m_1928144714898603219WordSection1"><p class="MsoNormal"><span style="font-size:11pt;font-family:"Calibri",sans-serif;color:rgb(31,73,125)" lang="EN-GB"><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:"Calibri",sans-serif;color:rgb(31,73,125)" lang="EN-GB"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:"Calibri",sans-serif;color:rgb(31,73,125)" lang="EN-GB">Cheers<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:"Calibri",sans-serif;color:rgb(31,73,125)" lang="EN-GB">Stefan<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:"Calibri",sans-serif;color:rgb(31,73,125)" lang="EN-GB"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:"Calibri",sans-serif;color:rgb(31,73,125)" lang="EN-GB"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-GB"><u></u> <u></u></span></p>
<p class="MsoNormal"><b><span style="font-size:11pt;font-family:"Calibri",sans-serif" lang="EN-US">From:</span></b><span style="font-size:11pt;font-family:"Calibri",sans-serif" lang="EN-US"> grass-dev [mailto:<a href="mailto:grass-dev-bounces@lists.osgeo.org" target="_blank">grass-dev-bounces@list<wbr>s.osgeo.org</a>]
<b>On Behalf Of </b>Rashad Kanavath<br>
<b>Sent:</b> mandag 5. februar 2018 17.06<br>
<b>To:</b> Moritz Lennert <<a href="mailto:mlennert@club.worldonline.be" target="_blank">mlennert@club.worldonline.be</a>><br>
<b>Cc:</b> <a href="mailto:grass-user@lists.osgeo.org" target="_blank">grass-user@lists.osgeo.org</a>; <a href="mailto:grass-dev@lists.osgeo.org" target="_blank">grass-dev@lists.osgeo.org</a>; Helmut Kudrnovsky <<a href="mailto:hellik@web.de" target="_blank">hellik@web.de</a>><br>
<b>Subject:</b> Re: [GRASS-dev] [GRASS-user] Keeping GRASS/OTB/... algorithm in qgis processing<u></u><u></u></span></p><div><div class="gmail-m_-6913107978359603392h5">
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<p class="MsoNormal">On Mon, Feb 5, 2018 at 2:49 PM, Moritz Lennert <<a href="mailto:mlennert@club.worldonline.be" target="_blank">mlennert@club.worldonline.be</a>> wrote:<u></u><u></u></p>
<blockquote style="border-color:currentcolor currentcolor currentcolor rgb(204,204,204);border-style:none none none solid;border-width:medium medium medium 1pt;padding:0cm 0cm 0cm 6pt;margin-left:4.8pt;margin-right:0cm">
<p class="MsoNormal">On 05/02/18 13:51, Helmut Kudrnovsky wrote:<u></u><u></u></p>
<blockquote style="border-color:currentcolor currentcolor currentcolor rgb(204,204,204);border-style:none none none solid;border-width:medium medium medium 1pt;padding:0cm 0cm 0cm 6pt;margin-left:4.8pt;margin-right:0cm">
<p class="MsoNormal">Von: "Moritz Lennert"<u></u><u></u></p>
<blockquote style="border-color:currentcolor currentcolor currentcolor rgb(204,204,204);border-style:none none none solid;border-width:medium medium medium 1pt;padding:0cm 0cm 0cm 6pt;margin-left:4.8pt;margin-right:0cm">
<p class="MsoNormal">I don't know how difficult it would be to create such algorithm<br>
descriptions automagically.<u></u><u></u></p>
</blockquote>
<p class="MsoNormal"> <a href="https://github.com/qgis/QGIS/search?p=1&q=processing+grass&type=&utf8=%E2%9C%93" target="_blank">https://github.com/qgis/QGIS<wbr>/search?p=1&q=processing+grass<wbr>&type=&utf8=%E2%9C%93</a><br>
<br>
AFAIU there are some general python scripts to provide it in processing:<br>
<br>
e.g.<br>
python/plugins/processing/algs<wbr>/grass7/Grass7AlgorithmProvide<wbr>r.py<u></u><u></u></p>
</blockquote>
<p class="MsoNormal" style="margin-bottom:12pt"><br>
No, this is the provider itself, not a tool to create the descriptions.<u></u><u></u></p>
<blockquote style="border-color:currentcolor currentcolor currentcolor rgb(204,204,204);border-style:none none none solid;border-width:medium medium medium 1pt;padding:0cm 0cm 0cm 6pt;margin-left:4.8pt;margin-right:0cm">
<p class="MsoNormal" style="margin-bottom:12pt"><br>
and there are a lot of txt files providing the module interface<br>
<br>
e.g<br>
python/plugins/processing/algs<wbr>/grass7/description/r.out.png.<wbr>txt<br>
r.out.png<br>
Export a GRASS raster map as a non-georeferenced PNG image<br>
Raster (r.*)<br>
QgsProcessingParameterRasterLa<wbr>yer|input|Input raster|None|False<br>
QgsProcessingParameterNumber|c<wbr>ompression|Compression level of PNG file (0 = none, 1 = fastest, 9 = best)|QgsProcessingParameterNu<wbr>mber.Integer|6|True|0|9<br>
<br>
and other files<u></u><u></u></p>
</blockquote>
<p class="MsoNormal"><br>
My question was whether it would be possible to create these description files more or less automagically.<br>
<br>
I think the python scripts for more complex operations have to be created manually.<br>
<br>
IIUC (from rapid reading of the threads on the qgis-developer list), Rashad's suggestion was to keep the *AlgorithmProvider code in the QGIS code base, but to possibly move the creation of the description and script files to a plugin managed outside QGIS core,
possibly by the respective external software teams.<u></u><u></u></p>
</blockquote>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Yes. your are right on track! the idea is external tools (processing providers) manage descriptor files in a format requested by qgis processing. <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">I see already name-of-grass-module --interface-descriptor which gives an xml for GRASS gui. <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">what qgis want is a csv in a specific format. The contents of qgis descriptor seems much less compared to --interface-descriptor.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Correct me if I am wrong, --interface-descriptor is available in all grass modules. So maybe a --qgis can do the work.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">This has some advantages.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">* GRASS developers are free to fix parameter name, parameter description, list of modules(add and remove) changes without affecting qgis.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">* grass 7.5 has 10 modules and grass 7.9 can have 15 and same QGIS will work.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">* QGIS will not need to maintain these files and keep updating/adding new modules with their release process. whatever is generated from a grass build/install have better integration with qgis<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">* Finally this descriptors for qgis are generated with a makefile target that allows users and packagers to include it. <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">* QGIS can use multiple version of grass by changing install prefix because descriptors are *always* found in a directory relative to install prefix.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">QGIS provider will be like:<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">I picked a descriptor file<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">parse and make the ui, <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">take input and execute whatever program the descriptor is to run.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">QGIS already manages parsing of parameters and running them. So for providers who wish to be integrated in qgis will deal with a descriptor file and things go fine.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Having a single interface to launch all or most of toolboxes (willing to contribute descriptor with installation) can have same way of execution. At that point, QGIS should consider<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">adding some generic code in provider and avoid plugins for such toolboxes.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<blockquote style="border-color:currentcolor currentcolor currentcolor rgb(204,204,204);border-style:none none none solid;border-width:medium medium medium 1pt;padding:0cm 0cm 0cm 6pt;margin-left:4.8pt;margin-right:0cm">
<p class="MsoNormal"><span style="color:rgb(136,136,136)"><br>
<br>
<span class="gmail-m_-6913107978359603392m_1928144714898603219m-8535335737944291886hoenzb">Moritz</span></span><u></u><u></u></p>
<div>
<div>
<p class="MsoNormal"><br>
______________________________<wbr>_________________<br>
grass-dev mailing list<br>
<a href="mailto:grass-dev@lists.osgeo.org" target="_blank">grass-dev@lists.osgeo.org</a><br>
<a href="https://lists.osgeo.org/mailman/listinfo/grass-dev" target="_blank">https://lists.osgeo.org/mailma<wbr>n/listinfo/grass-dev</a><u></u><u></u></p>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal"><br>
<br clear="all">
<u></u><u></u></p>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<p class="MsoNormal">-- <u></u><u></u></p>
<div>
<div>
<p class="MsoNormal"><span style="font-family:"Arial",sans-serif">Regards,<br>
Rashad</span><u></u><u></u></p>
</div>
</div>
</div>
</div>
</div></div></div>
</div>
<br>______________________________<wbr>_________________<br>
grass-user mailing list<br>
<a href="mailto:grass-user@lists.osgeo.org" target="_blank">grass-user@lists.osgeo.org</a><br>
<a href="https://lists.osgeo.org/mailman/listinfo/grass-user" rel="noreferrer" target="_blank">https://lists.osgeo.org/mailma<wbr>n/listinfo/grass-user</a><br></blockquote></div><br></div></div></div></div></div>