[QGIS-Developer] QGIS 3, the state of Processing and GRASS 7

[Stephen]

Hi Nyall,

thanks for your very thoughtful reply!

On 28.05.2018 06:55, Nyall Dawson wrote:
> On 27 May 2018 at 08:11, Stephen <posting at vodacomm.ca> wrote:
>> Hello everyone,
>>
>> for weeks now I have been trying to port standalone PyQGIS code that
>> makes extensive use of Processing from QGIS 2.18 to QGIS 3. I'm running
>> 3.0.2, the most recent stable version available on Gentoo Linux.
>>
>> The port was motivated by the need to have code that would work in the
>> medium term (that was "future ready", so to speak) and the fact that
>> packages requiring the PyQt4 libraries are being progressively removed
>> from the package tree of my distribution (for a variety of good
>> reasons). QGIS 2.x is among them. Maintaining my system in a usable
>> state while also retaining QGIS 2.x ultimately became untenable.
> 
> Hi Stephen,
> 
> Sorry to hear about your experiences with 3.0 -- they certainly sound
> frustrating.
> 
> Unfortunately there's no easy answers here. It's a difficult choice
> between sticking with the recommended LTR enterprise release 2.18,
> which is stable and has had LOTS of testing, versus targeting 3.x and
> ensuring code is future proof. Generally the project's advice is for
> mission critical applications, stick with 2.18 until the next LTR
> release (3.4, later this year). That's why 3.0 has a giant "early
> adopter" warning banner on the splash screen.>
> What I definitely wouldn't recommend doing is targeting 3.0 and not
> 3.2 -- 3.0 has already seen its final release (3.0.3) and is
> effectively end-of-life'd now. So if you decide to target your
> application at 3.x, better aim for 3.2 at least.

That all makes sense; I had only switched back to 3.0.2 because my
previously working parts of my code had stopped working in 3.1 master; I
felt like I was actually going backwards.

I think the best short-term approach is to try, somehow, to get a
working 2.18 environment on my system and revert to that working code.

The work I've done on the QGIS 3 code isn't lost (I'm using git :) )

Then I can sensibly aim for making it work with 3.2.

> 
> You're also in the difficult position of relying on the processing
> SAGA/grass providers. Unfortunately both these providers are
> under-maintained (vs say the native QGIS provider and the GDAL/OGR
> provider, which are heavily unit tested and generally very stable). If
> you can, I'd always recommend using a native QGIS algorithm or
> GDAL/OGR algorithm in place of GRASS or SAGA (for now at least).

You are speaking my language!

My only hesitancy: I haven't thoroughly checked, but I can't be sure
that all the GRASS and SAGA algorithms used here have QGIS and GDAL/OGR
counterparts.

But yes, perhaps the best overall approach is to bet on GDAL/OGR and
native QGIS.

> So with that said, let's try and resolve your issues...

<snip>

>> For an example of what I am dealing with, have a look at this post on
>> the qgis-user list:
>>
>> https://lists.osgeo.org/pipermail/qgis-user/2018-May/042461.html
>>
>> (I "fixed" this by editing Grass7Utils.py to replace the call to
>> Grass7Utils.command with Grass7Utils.grassBin().)
>>
> 
> I'll reply separately on that thread to keep the conversation together.

Thanks!

>> Or consider this bit of code:
>> ...
>> The corresponding parameter check in r_null.py is here:
>>
>> https://github.com/qgis/QGIS/blob/307d082e3de804064a7295aa079ee6cd0e47845a/python/plugins/processing/algs/grass7/ext/r_null.py#L29
>>
>> So on *success*, this parameter check returns None, None, which causes
>> Processing to fail with "Unable to execute algorithm"...
> 
> Definitely a bug - there's a fix and unit test protecting against this
> behavior with providers at https://github.com/qgis/QGIS/pull/7095.

I didn't see much sense in filing a bug report before determining
whether this problem had already been identified :)

Glad to hear I'm not imagining things, though.

>> But my ported code has to be working by Tuesday or I have to pull the
>> ripcord on the entire project. The only other possibility I can see at
>> this point is to revert back to my working 2.x code and establish some
>> 2.x 'island' on which it can run. Honestly, I dread this. How can that
>> be in the interest of all the QGIS volunteers?
>>
>> Say you had an project like this that used lots of calls to Processing
>> and relied on external algorithms from GRASS 7 and SAGA, what would you
>> do? What approach would you take? What can I do to make this port
>> successful?
>>
> 
> Not sure if we can get master in shape by Tuesday (given that 3.2
> release isn't still for a couple of weeks), and in any case you'd need
> to run master and eventually 3.2 to get these fixes (since 3.0 won't
> have any more patch releases). But if you had a bit more breathing
> room, I'd like to think we can resolve all your issues before 3.2
> final release....

As above, let's not abandon the idea of making my project 3.2 compatible!

I will set up an 2.18 environment (if I can do that without totally
breaking my system ;-) ) to take some of the pressure off in the
immediate term.

Cheers

Stephen