[QGIS-Developer] QGIS 3, the state of Processing and GRASS 7

[Nyall Dawson]

On 27 May 2018 at 08:11, Stephen <posting at vodacomm.ca> wrote:
> Hello everyone,
>
> for weeks now I have been trying to port standalone PyQGIS code that
> makes extensive use of Processing from QGIS 2.18 to QGIS 3. I'm running
> 3.0.2, the most recent stable version available on Gentoo Linux.
>
> The port was motivated by the need to have code that would work in the
> medium term (that was "future ready", so to speak) and the fact that
> packages requiring the PyQt4 libraries are being progressively removed
> from the package tree of my distribution (for a variety of good
> reasons). QGIS 2.x is among them. Maintaining my system in a usable
> state while also retaining QGIS 2.x ultimately became untenable.

Hi Stephen,

Sorry to hear about your experiences with 3.0 -- they certainly sound
frustrating.

Unfortunately there's no easy answers here. It's a difficult choice
between sticking with the recommended LTR enterprise release 2.18,
which is stable and has had LOTS of testing, versus targeting 3.x and
ensuring code is future proof. Generally the project's advice is for
mission critical applications, stick with 2.18 until the next LTR
release (3.4, later this year). That's why 3.0 has a giant "early
adopter" warning banner on the splash screen.

What I definitely wouldn't recommend doing is targeting 3.0 and not
3.2 -- 3.0 has already seen its final release (3.0.3) and is
effectively end-of-life'd now. So if you decide to target your
application at 3.x, better aim for 3.2 at least.

You're also in the difficult position of relying on the processing
SAGA/grass providers. Unfortunately both these providers are
under-maintained (vs say the native QGIS provider and the GDAL/OGR
provider, which are heavily unit tested and generally very stable). If
you can, I'd always recommend using a native QGIS algorithm or
GDAL/OGR algorithm in place of GRASS or SAGA (for now at least).

So with that said, let's try and resolve your issues...

> The porting work is complicated by the lack of meaningful error messages
> when calls to external algorithms fail (either it returns with exit code
> 1 or I get "Unable to execute algorithm"). The only way I have been able
> to make any progress at all has been through deep forensic work in the
> debugger. This process is very slow when it works at all. I spend
> minutes to hours debugging each algorithm call, and then the next one
> comes along, fails with similarly uninformative errors, and the process
> starts all over again.
>
> For an example of what I am dealing with, have a look at this post on
> the qgis-user list:
>
> https://lists.osgeo.org/pipermail/qgis-user/2018-May/042461.html
>
> (I "fixed" this by editing Grass7Utils.py to replace the call to
> Grass7Utils.command with Grass7Utils.grassBin().)
>

I'll reply separately on that thread to keep the conversation together.

> Or consider this bit of code:
> ...
> The corresponding parameter check in r_null.py is here:
>
> https://github.com/qgis/QGIS/blob/307d082e3de804064a7295aa079ee6cd0e47845a/python/plugins/processing/algs/grass7/ext/r_null.py#L29
>
> So on *success*, this parameter check returns None, None, which causes
> Processing to fail with "Unable to execute algorithm"...

Definitely a bug - there's a fix and unit test protecting against this
behavior with providers at https://github.com/qgis/QGIS/pull/7095.

> But my ported code has to be working by Tuesday or I have to pull the
> ripcord on the entire project. The only other possibility I can see at
> this point is to revert back to my working 2.x code and establish some
> 2.x 'island' on which it can run. Honestly, I dread this. How can that
> be in the interest of all the QGIS volunteers?
>
> Say you had an project like this that used lots of calls to Processing
> and relied on external algorithms from GRASS 7 and SAGA, what would you
> do? What approach would you take? What can I do to make this port
> successful?
>

Not sure if we can get master in shape by Tuesday (given that 3.2
release isn't still for a couple of weeks), and in any case you'd need
to run master and eventually 3.2 to get these fixes (since 3.0 won't
have any more patch releases). But if you had a bit more breathing
room, I'd like to think we can resolve all your issues before 3.2
final release....

Nyall