[Qgis-developer] QGIS Processing Framework

[Camilo Polymeris]

On Wed, Apr 27, 2011 at 2:24 PM, Julien Malik <julien.malik at c-s.fr> wrote:
> Hi Camilo, Hi all,
>
> First of all congratulation for your GSoC project !
>

Thanks!

> For OTB we are developing a "module" framework also, with auto-generated
> interface. As I told people at the HF, we have made a first step with our
> last 3.8 release, but are not satisfied enough with the GUI part, so we are
> currently revamping the core to add a lot of functionnalities.
>
> Our ideas/wish-list is summed up here :
> http://wiki.orfeo-toolbox.org/index.php/OTB-Wrapper
>
> The current code for what we want to push into QGis is here (please note
> that it is in development, i've completed the first usefull app just after
> the HackFest) :
> http://hg.orfeo-toolbox.org/OTB-Wrapper

Looks very good, and in principle similar to what I was doing (or
trying to do) with SAGA, see github repo. But, from what I can tell,
is very OTB-specific. I am not sure how we can integrate this into a
common framework.

> [1]
> We should be very careful about not being limited by "too much genericity".
>
> The SAGA and GRASS modules have each their own design/parameter types/inner
> but also outer logic.
> On the OTB side, we have some liberty since we are currently working on
> them, but we have some plans to integrate them in other tools, possibly
> benefiting from their additional possibilities (the pipeline capabilities of
> Monteverdi for example).
> We have some ideas for making the GUI as dynamic as possible (and it is with
> their Qt/Qgis implementation that we plan to have the most full-featured
> GUI) but I guess other libs will have capabilities that we don't have.

I feel in general QGIS should handle the GUI part.
That makes the framework usable in non-GUI contexts and only dependant
on QtCore (the non-QGIS side, I mean). In my test code I used a
QFormLayout, populated with Widgets according to the type of the
parameters alone. (screenshot linked from the "QGIS Processing
Framework" page)
I imagined the particular implementations using the existing Plugin
mechanism and thus having access to QgisInterface anyway, if really
necessary.

> Trying to fit everyone in the same framework will be hard : either each
> library (SAGA/GRASS/OTB) will only use a subset of the provided
> functionnalities, either we will end up taking the common factor of all libs
> and everyone will feel limited. I'm worried than we fall into one of these
> two pitfalls. Please do not impose to much constraints !
>
> Reactions very welcome on this point !

IMO, it is this genericity that would enable the different
implementations to talk to each other... inter-library pipelining
being possible if the parameter (including I/O) types match.

Compare the list of parameters for OTB and SAGA:
https://github.com/polymeris/qgis/wiki/List-of-parameters
I think a common ground can be found for most cases, while retaining
the ability to specify implementation-specific types.
That allows consistency in the user interface and easier scripting (no
need to worry about how each implementation handles each data type).

> What seems reasonable to me at first sight :
> - using similar widgets for similar functionnalities/parameter types, maybe
> with base classes to reuse (but again make them really high level !).
> - having a common way to call the modules
> - all in all, make them all look the same
> This will give the user the illusion that they all comes from the same
> library, but will give each implementation the necessary liberty to fine
> tune their GUIs using the specificities of their library/internal "module"
> framework.
>
> We have the chance of being in a similar phase of development for both
> project, so i'm sure we will find the right common ground.
>

Agreed. The framework shouldn't impose any constraints. While it can
provide validators, data types, predefined tags, widgets, etc., each
implementation is free to create their own, or even talk directly to
QGIS, bypassing the framework, when necessary.

>
> [2]
> Chaining modules from different libraries seems a little ambitious to me,
> but this would be awesome !
> As you said Camilo, I think we can leave pipelining out as a first step.
> We are not really focusing on that part now neither, though the current
> implementation should allow this.

I think chaining modules is possible if input/ouput formats are well
defined. At least in a crude "execute module A, convert data, execute
module B" way.

>
> [3]
> You seem to consider parameter validation only in an independent way
>
> From our experience it is insufficient (for example, absolute default values
> is not enough).
> How we plan to solve this (trying to keep it simple also...), is to give our
> modules 3 main entry points :
> - description of the parameters
> - a global method to update all the parameters, called as soon as one
> parameter is changed
> - an execution method
> Let's imagine a simple orthorectification application with the following
> params :
> - input/output image
> - the output CRS
> - output origin, size, and spacing
> We want the origin, size and spacing to be auto-computed each time the input
> or CRS changes
> If CRS changes from UTM to WGS84, we expect the outputorigin units/display
> to also change from meter to degree, for example.

Yes, very important point. One possibility would be to use signals to
connect the framework to each parameter's validator (to change
constraints) and to the module (to add/remove parameters, as per [4])

> [4] (related to [3])
> Our first working-and-usefull application in this new framework we're
> building shows also dynamic parameters list in the GUI (see the screenshot
> on the Wiki).
> It is a smoothing application with 3 modes: Median, Gaussian, Anisotropic
> Filtering.
> Each mode has associated params. Median and Gaussian needs a radius,
> Anisotropic Filtering needs a time step and a number of iterations.
> So below the combobox, the widget displayed depend on the combobox value.
>
> [5]
> The model we imagined for the parameter list will be a tree more than a flat
> list.
> A group of parameters is just one node of the tree, containing a parameters
> list, where each parameter can also be a group of parameters.
>

Ok. A generic list type could be defined as parameter type. Perhaps
implemented with QList<Parameter> (and add that to the QVariant union,
of course). Not to be confused with list-of-something parameters.

> We would also want checkable parameters.
> mandatory parameters should have a value.
> non-mandatory parameters would have a checkbox to be activated.

That shouldn't be too hard. I think it is ok to have the validator
handle that, accepting empty values if the parameter is optional.
It is a minor detail, but from a UI point of view, I wouldn't use
checkboxes, because it demands extra clicks from the end-user, but
rather use some icon and/or a bold typeface to identifiy mandatory
parameters. This sort of thing would have to be checked with people
more versed in UI design, though.

>
> Hierarchical tags seems very reasonable to me.
>
>>  The names of tags will be
>> left for developers' creativity.
>
> A list of default available tags should be provided.
> If the developer feel it can't fit in those, then it can create a new tag.
>

Agreed.

>> Formats and format conversion: for import and export of map layers the
>> framework should support any layer loaded (or loadable) by QGIS. In
>> case the library uses a different format for input/output, it should
>> take care of import/export.
>
> I'm not sure this can be handled in the framework, but more in the different
> implementations.
> The input images to our modules will be OTB specific data objects, and the
> transformation from QGis raster (= filename) to OTB raster would be done the
> OTB-QGis plugins implementation (you can check the current Smoothing
> application code).

I think both the framework can provide a list of formats it
understands, and which it prefers (to avoid unnecessary conversion).
If the implementation understands one of those, no problem. Else, it
is up to the implementation to convert.

As with widgets, tags, and everything else: The framework provides
basic options, without forcing the implementation to use them.

>> Feedback: many modules are able to return some feedback - either in
>> the form of a progress to indicate how long the process will take - or
>> to return some messages that will be shown to user. Support for
>> cancelling a running module would be helpful too, but not really
>> necessary right now.
>
> Must-have also !
>
> Having a common framework to report progress and log messages would be
> great.
> If its in Qt, it should be through signals.
> I already saw some code making a bridge between the OTB events framework and
> Qt signals, I'd like to reuse this.
>

Agreed. Actually, from the framework's point of view, events and
progress indicators and such can just be output parameters. And the
same mechanism that is used to update input parameters (for
validation, [3]) can be used to report back.

Well. Thanks, Julien, for your comments. You have a very clear picture
of the issues and limitations involved in this project and I really
appreciate the time you took in sharing it with us.
Also, I think in general we agree. Except, as I said before, I would
move as much of the GUI-related code from the implementation side to
the QGIS side to make it more-scripting friendly and consistent.

Of course, I am only starting with this and everything I said above
may very well be wrong or irrelevant. Please correct/criticize.

Regards,
Camilo