[Qgis-developer] QGIS Processing Framework

[Julien Malik]

Hi Camilo, Hi all,

First of all congratulation for your GSoC project !

Some apologies for the delay in my response, we are approaching the next 
OTB release so I'm busy on fixing little stuff/builds here and there. 
Please don't expect real time answers in the next week or two.
But it's nice to see you left the starting blocks a while ago :)

For OTB we are developing a "module" framework also, with auto-generated 
interface. As I told people at the HF, we have made a first step with 
our last 3.8 release, but are not satisfied enough with the GUI part, so 
we are currently revamping the core to add a lot of functionnalities.

Our ideas/wish-list is summed up here :
http://wiki.orfeo-toolbox.org/index.php/OTB-Wrapper

The current code for what we want to push into QGis is here (please note 
that it is in development, i've completed the first usefull app just 
after the HackFest) :
http://hg.orfeo-toolbox.org/OTB-Wrapper
Don't hesitate to ask me for help to build it if you need.


Some general comments :

[1]
We should be very careful about not being limited by "too much genericity".

The SAGA and GRASS modules have each their own design/parameter 
types/inner but also outer logic.
On the OTB side, we have some liberty since we are currently working on 
them, but we have some plans to integrate them in other tools, possibly 
benefiting from their additional possibilities (the pipeline 
capabilities of Monteverdi for example).
We have some ideas for making the GUI as dynamic as possible (and it is 
with their Qt/Qgis implementation that we plan to have the most 
full-featured GUI) but I guess other libs will have capabilities that we 
don't have.

Trying to fit everyone in the same framework will be hard : either each 
library (SAGA/GRASS/OTB) will only use a subset of the provided 
functionnalities, either we will end up taking the common factor of all 
libs and everyone will feel limited. I'm worried than we fall into one 
of these two pitfalls. Please do not impose to much constraints !

Reactions very welcome on this point !

What seems reasonable to me at first sight :
- using similar widgets for similar functionnalities/parameter types, 
maybe with base classes to reuse (but again make them really high level !).
- having a common way to call the modules
- all in all, make them all look the same
This will give the user the illusion that they all comes from the same 
library, but will give each implementation the necessary liberty to fine 
tune their GUIs using the specificities of their library/internal 
"module" framework.

We have the chance of being in a similar phase of development for both 
project, so i'm sure we will find the right common ground.

By the way, after SAGA, Grass and OTB, maybe someone will consider Ossim 
: they also have a similar framework for modules with autogenerated 
interfaces, to do raster processing.
Seeing Maxim presenting a Qgis classification plugin based on OpenCV, 
maybe you will also end up with some OpenCV modules in the future...

[2]
Chaining modules from different libraries seems a little ambitious to 
me, but this would be awesome !
As you said Camilo, I think we can leave pipelining out as a first step.
We are not really focusing on that part now neither, though the current 
implementation should allow this.


[3]
You seem to consider parameter validation only in an independent way

 From our experience it is insufficient (for example, absolute default 
values is not enough).
How we plan to solve this (trying to keep it simple also...), is to give 
our modules 3 main entry points :
- description of the parameters
- a global method to update all the parameters, called as soon as one 
parameter is changed
- an execution method

Let's imagine a simple orthorectification application with the following 
params :
- input/output image
- the output CRS
- output origin, size, and spacing
We want the origin, size and spacing to be auto-computed each time the 
input or CRS changes
If CRS changes from UTM to WGS84, we expect the outputorigin 
units/display to also change from meter to degree, for example.

[4] (related to [3])
Our first working-and-usefull application in this new framework we're 
building shows also dynamic parameters list in the GUI (see the 
screenshot on the Wiki).
It is a smoothing application with 3 modes: Median, Gaussian, 
Anisotropic Filtering.
Each mode has associated params. Median and Gaussian needs a radius, 
Anisotropic Filtering needs a time step and a number of iterations.
So below the combobox, the widget displayed depend on the combobox value.

[5]
The model we imagined for the parameter list will be a tree more than a 
flat list.
A group of parameters is just one node of the tree, containing a 
parameters list, where each parameter can also be a group of parameters.



Please find also some other more specific comments below :




Le 27/04/2011 11:20, Martin Dobias a écrit :
> Hi Camilo
>
> On Tue, Apr 26, 2011 at 9:46 PM, Camilo Polymeris<cpolymeris at gmail.com>  wrote:
>> Hello everyone,
>> I was accepted as a student in GSoC to implement a "SAGA interface for QGIS".
> First of all congratulations to you and to other accepted students for
> the Summer of Code programme!
>
>> There are also proyects that try to interface QGIS with GRASS and the
>> Orfeo Toolbox (OTB). Paolo has informed me that the creation of a
>> common framework for those systems was discussed and approved during
>> HF. A very interesting idea, IMO, that completely changes the way I
>> would have to work on my project.
> Great that you like the idea of generic framework for processing tools
> in QGIS. Helping us to bring this framework from ideas into code would
> be another great benefit of your project.
>
>
>> While I didn't participate in those discussions and don't know what
>> conclusions were drawn, as a brainstorming exercise I am writing a
>> short description of what a very simple API to that framework could
>> look like:
>>
>> https://github.com/polymeris/qgis/wiki/QGIS-Processing-Framework
>>
>> Would that more or less fit what you have in mind? Please comment/criticize.
> The discussion was not going into implementation details, we were
> mostly collecting the ideas how the things should work.
>
> Framework implementation: as you write in the proposal, ideally it
> should be implemented in C++ with Python bindings. I would suggest you
> to start the implementation in Python and skip C++ coding at least at
> the beginning. My experience is that Python is great for this kind of
> prototyping (no compiling, shorter code etc). If you will end up with
> sufficient amount of time at the end of the summer, you can move that
> into C++ and add Python bindings. The idea is not to waste time with
> writing and rewriting various c++ classes when the API is not clear
> yet. Consider this as a suggestion, not a command how to do it :-)
>
> Forms/Parameters: I would start with with generic input fields (line
> edit / spin box) for integers, floats, strings that can be constrained
> with a range (numbers) or regular expressions (strings) - easily
> achieved with QValidator classes. Then some custom widgets can be
> added for better handling of some types: layer selector, enumeration
> etc. The framework may be extensible so that if a library often uses a
> particular custom type, the plugin integrating the library could add a
> custom widget for it.

Please see [3]
I fear it will be too limiting to consider parameters only independently


> Advanced parameters: some modules may allow users set lots of
> parameters. Although this is handy, many times the default values work
> well and only few basic parameters are changed regularly. It should be
> possible to mark some parameters as advanced and they would be hidden
> by default, they would show up after clicking a checkbox or a button.
> For example, when creating a vector buffer, the distance would be a
> basic parameter while the number of segments per quadrant would be
> advanced parameter.

Yes basic/advanced parameters seems a must-have to me.
We want not experienced users to use the modules, but we surely want to 
give all the possible finetuning possibilities to those who "know what 
they're doing"


We would also want checkable parameters.
mandatory parameters should have a value.
non-mandatory parameters would have a checkbox to be activated.



> Categorization: currently we are starting to face some trouble with
> organization of plugins and the "modules" they provide. We thought
> about creating fixed categories as in GRASS (e.g. Raster, Vector,
> Database) where the modules would be placed. We also considered custom
> categories and subcategories. The general problem with categories is
> that many modules do not suit them well if they are not coming from a
> well-organized library such as grass, saga or otb.
> The conclusion that I recall from the discussion at hackfest is that
> tagging is probably the only way to go: each module could have several
> tags assigned to it, then the list of modules would be a tree with top
> level items containing tags and modules would be shown inside these
> tags. Bigger libraries requiring deeper hierarchy might come with
> hierarchical tags like "SAGA/Grid - Tools".
Hierarchical tags seems very reasonable to me.

>   The names of tags will be
> left for developers' creativity.

A list of default available tags should be provided.
If the developer feel it can't fit in those, then it can create a new tag.

> This list of processing modules should later get some advanced
> functionality like searching, favourite modules or even attaching most
> used modules on a toolbar, but that is something for the future, we
> are not expecting you to do that during the summer :-) A simple tree
> listing tags and modules assigned to them will suffice.
Agreed.

> Formats and format conversion: for import and export of map layers the
> framework should support any layer loaded (or loadable) by QGIS. In
> case the library uses a different format for input/output, it should
> take care of import/export.

I'm not sure this can be handled in the framework, but more in the 
different implementations.
The input images to our modules will be OTB specific data objects, and 
the transformation from QGis raster (= filename) to OTB raster would be 
done the OTB-QGis plugins implementation (you can check the current 
Smoothing application code).

> Running modules from scripts: you probably have that in mind but let's
> keep it explicit: the framework should provide API that would allow
> user to run modules from python console or from a plugin, QGIS would
> basically act just as a default GUI using that API. This will finally
> allow users to run the processing modules in batches - poor man's
> pipelining :-)
>
> Feedback: many modules are able to return some feedback - either in
> the form of a progress to indicate how long the process will take - or
> to return some messages that will be shown to user. Support for
> cancelling a running module would be helpful too, but not really
> necessary right now.

Must-have also !

Having a common framework to report progress and log messages would be 
great.
If its in Qt, it should be through signals.
I already saw some code making a bridge between the OTB events framework 
and Qt signals, I'd like to reuse this.


>
>
> Regards
> Martin
> _______________________________________________
> Qgis-developer mailing list
> Qgis-developer at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/qgis-developer
>