[QGIS-Developer] QGIS 3, the state of Processing and GRASS 7

[Stephen]

Hello everyone,

for weeks now I have been trying to port standalone PyQGIS code that
makes extensive use of Processing from QGIS 2.18 to QGIS 3. I'm running
3.0.2, the most recent stable version available on Gentoo Linux.

The port was motivated by the need to have code that would work in the
medium term (that was "future ready", so to speak) and the fact that
packages requiring the PyQt4 libraries are being progressively removed
from the package tree of my distribution (for a variety of good
reasons). QGIS 2.x is among them. Maintaining my system in a usable
state while also retaining QGIS 2.x ultimately became untenable.

The porting work is complicated by the lack of meaningful error messages
when calls to external algorithms fail (either it returns with exit code
1 or I get "Unable to execute algorithm"). The only way I have been able
to make any progress at all has been through deep forensic work in the
debugger. This process is very slow when it works at all. I spend
minutes to hours debugging each algorithm call, and then the next one
comes along, fails with similarly uninformative errors, and the process
starts all over again.

For an example of what I am dealing with, have a look at this post on
the qgis-user list:

https://lists.osgeo.org/pipermail/qgis-user/2018-May/042461.html

(I "fixed" this by editing Grass7Utils.py to replace the call to
Grass7Utils.command with Grass7Utils.grassBin().)

Or consider this bit of code:

>    processing.run("grass7:r.null", {'map': tempbuffer,
>                                      'null': 0,
>                                      'GRASS_REGION_PARAMETER': extent,
>                                      'GRASS_REGION_CELLSIZE_PARAMETER': 0.008333,
>                                      'output': file_buffer},
>                    feedback=QgsProcessingFeedback())

This fails with "Unable to execute algorithm", which is thrown here:

https://github.com/qgis/QGIS/blob/307d082e3de804064a7295aa079ee6cd0e47845a/python/plugins/processing/core/Processing.py#L141

when I step through this method, I end up at

https://github.com/qgis/QGIS/blob/307d082e3de804064a7295aa079ee6cd0e47845a/python/plugins/processing/algs/grass7/Grass7Algorithm.py#L953

The corresponding parameter check in r_null.py is here:

https://github.com/qgis/QGIS/blob/307d082e3de804064a7295aa079ee6cd0e47845a/python/plugins/processing/algs/grass7/ext/r_null.py#L29

So on *success*, this parameter check returns None, None, which causes
Processing to fail with "Unable to execute algorithm"...

It seems like everyone is working so hard to make QGIS 3 a success, and
I want to contribute to that success by implementing applications using
QGIS 3 and helping test.

But my ported code has to be working by Tuesday or I have to pull the
ripcord on the entire project. The only other possibility I can see at
this point is to revert back to my working 2.x code and establish some
2.x 'island' on which it can run. Honestly, I dread this. How can that
be in the interest of all the QGIS volunteers?

Say you had an project like this that used lots of calls to Processing
and relied on external algorithms from GRASS 7 and SAGA, what would you
do? What approach would you take? What can I do to make this port
successful?

Thanks and regards

Stephen Bosch