[QGIS-Developer] FME to QGIS functionality mapping

[Nyall Dawson]

On Tue, 7 Apr 2020 at 17:35, Andreas Neumann <a.neumann at carto.net> wrote:
>
> Hi Nyall,
>
> Thank you for sharing this interesting document.
>
> I have questions and remarks:
>
> 1. One fundamental difference between FME and QGIS modeler, as far as I
> understand it, is the fact that FME data sources are much more tightly
> coupled with the model than in QGIS modeler. This has the advantage,
> that the data readers, transformers and writers always know what
> geometry types and attributes they have. The disadvantage is that it is
> harder to exchange data sources with sources that have other attributes.
> In my experience the FME approach is what most users prefer. They like
> the fact that they can always see at each transformer what attributes
> are available at this stage. Would you see an improvement - perhaps
> another modeler mode - where also QGIS modeler would be more tightly
> coupled with the data sources and that the model and algorithms would
> always know at any step what attributes are available at this stage? Or
> is this a fundamental difference between the two approaches that
> couldn't be bridged?

I think it's totally possible -- we just need to know how each
algorithm transforms its inputs into an output data set in advance
(i.e. the crs, wkb type, and fields).  API which permits this was
recently added, but it's not yet hooked into the modeler.  There's
still considerable work to do for that, but it's far from impossible!

> 2. One aspect where FME shines is the ability to graphically debug
> models. One can at every step in the model get the view (map and table)
> of the intermediate step. One can also see at each branch in the model
> how many features take this route and how many the other route. I know
> this doesn't have anything to do with the transformer list you shared,
> but this an aspect that I think, many users like about FME.

A recent improvement is that after running a model in the designer,
you can now see the actual values of inputs and outputs which were
used for each child algorithm in the model. It's a partial step
towards the FME approach. To get the rest of the way we'd need:
1. to ensure that every algorithm has numeric outputs for the number
of features processed
2. ideally to allow the model to run on a background thread inside the
designer, so that you'd see these input/output values populate
instantly as the model runs (would also allow us to color-code child
algorithms by state, e.g. completed/running/pending/skipped).

1. is pretty straightforward, just a lot of grunt work. 2. is more
complex (and I suspect only achievable in builds based on qt 5.10+
(since we'd need to invoke the child algorithm prepare steps on the
main thread)... but its probably time to bump the minimum anyway).

Anyway, definitely achievable in the near future.

> I agree that there would a lot of "low hanging" fruits and I'm looking
> forward to more feature parity between the two. And to the discussion
> around it.

Me too! When 3.14 is released the model capabilities in QGIS will be
much much improved vs previous releases. Just the fact that **every**
parameter for child algorithms (including input and output layers) can
now be defined using qgis expressions opens up a WHOLE new world of
possibilities alone...

Nyall



>
> Greetings,
>
> Andreas
>
> Am 07.04.20 um 03:27 schrieb Nyall Dawson:
> > Hi list,
> >
> > Just wanted to give a heads up on a document I've recently completed,
> > which maps existing FME functionality across to the equivalent QGIS
> > Processing or Expression equivalent:
> >
> > https://github.com/nyalldawson/qgis-processing-hitlist/blob/master/fme.md
> >
> > Since FME transformers are both algorithm-like and expression-like
> > functionality, the document splits the QGIS equivalents into columns
> > for both algorithms and expressions. Wherever the transformer
> > functionality doesn't apply, a "N/A" is entered in the corresponding
> > column (e.g. the "StringCaseChanger" transformer doesn't make sense
> > the implement as a QGIS Processing algorithm -- it's equivalent is
> > instead the expression functions "lower", "upper" and "title". And
> > conversely the "DuplicateFilter" transformer maps to the Processing
> > algorithms "Delete Duplicate Geometries" and "Delete duplicates by
> > attribute", but doesn't make sense to have an expression function
> > equivalent!)
> >
> > There's also Transformers in FME which really don't need an equivalent
> > in QGIS -- e.g. those relating to styling features.
> >
> > Overall we have a good level of coverage of the most important
> > functionality. There's a LOT of low-hanging fruit which could easily
> > be addressed in small PRs (e.g. things like the
> > "CharacterCodeExtractor" transformer would be a trivial task to
> > implement as a new QGIS expression function ... a GREAT task for
> > someone getting started with QGIS development!).
> >
> > (and just for completeness - QGIS offers hundreds of processing
> > algorithms and expression functionality which **doesn't** have
> > equivalents in FME. I suspect the reverse document would have just as
> > many "missing" cells!)
> >
> > Feedback welcome! At some stage I think it would be nice to move this
> > document across to the QGIS repo, and possibly pull into the QGIS
> > documentation itself (along with similar documents for Arc/...
> > functionality!)
> >
> > Nyall
> > _______________________________________________
> > QGIS-Developer mailing list
> > QGIS-Developer at lists.osgeo.org
> > List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
> > Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer
> _______________________________________________
> QGIS-Developer mailing list
> QGIS-Developer at lists.osgeo.org
> List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer