[Qgis-developer] QGIS Processing Framework
julien.malik at c-s.fr
Wed Apr 27 14:24:25 EDT 2011
Hi Camilo, Hi all,
First of all congratulation for your GSoC project !
Some apologies for the delay in my response, we are approaching the next
OTB release so I'm busy on fixing little stuff/builds here and there.
Please don't expect real time answers in the next week or two.
But it's nice to see you left the starting blocks a while ago :)
For OTB we are developing a "module" framework also, with auto-generated
interface. As I told people at the HF, we have made a first step with
our last 3.8 release, but are not satisfied enough with the GUI part, so
we are currently revamping the core to add a lot of functionnalities.
Our ideas/wish-list is summed up here :
The current code for what we want to push into QGis is here (please note
that it is in development, i've completed the first usefull app just
after the HackFest) :
Don't hesitate to ask me for help to build it if you need.
Some general comments :
We should be very careful about not being limited by "too much genericity".
The SAGA and GRASS modules have each their own design/parameter
types/inner but also outer logic.
On the OTB side, we have some liberty since we are currently working on
them, but we have some plans to integrate them in other tools, possibly
benefiting from their additional possibilities (the pipeline
capabilities of Monteverdi for example).
We have some ideas for making the GUI as dynamic as possible (and it is
with their Qt/Qgis implementation that we plan to have the most
full-featured GUI) but I guess other libs will have capabilities that we
Trying to fit everyone in the same framework will be hard : either each
library (SAGA/GRASS/OTB) will only use a subset of the provided
functionnalities, either we will end up taking the common factor of all
libs and everyone will feel limited. I'm worried than we fall into one
of these two pitfalls. Please do not impose to much constraints !
Reactions very welcome on this point !
What seems reasonable to me at first sight :
- using similar widgets for similar functionnalities/parameter types,
maybe with base classes to reuse (but again make them really high level !).
- having a common way to call the modules
- all in all, make them all look the same
This will give the user the illusion that they all comes from the same
library, but will give each implementation the necessary liberty to fine
tune their GUIs using the specificities of their library/internal
We have the chance of being in a similar phase of development for both
project, so i'm sure we will find the right common ground.
By the way, after SAGA, Grass and OTB, maybe someone will consider Ossim
: they also have a similar framework for modules with autogenerated
interfaces, to do raster processing.
Seeing Maxim presenting a Qgis classification plugin based on OpenCV,
maybe you will also end up with some OpenCV modules in the future...
Chaining modules from different libraries seems a little ambitious to
me, but this would be awesome !
As you said Camilo, I think we can leave pipelining out as a first step.
We are not really focusing on that part now neither, though the current
implementation should allow this.
You seem to consider parameter validation only in an independent way
From our experience it is insufficient (for example, absolute default
values is not enough).
How we plan to solve this (trying to keep it simple also...), is to give
our modules 3 main entry points :
- description of the parameters
- a global method to update all the parameters, called as soon as one
parameter is changed
- an execution method
Let's imagine a simple orthorectification application with the following
- input/output image
- the output CRS
- output origin, size, and spacing
We want the origin, size and spacing to be auto-computed each time the
input or CRS changes
If CRS changes from UTM to WGS84, we expect the outputorigin
units/display to also change from meter to degree, for example.
 (related to )
Our first working-and-usefull application in this new framework we're
building shows also dynamic parameters list in the GUI (see the
screenshot on the Wiki).
It is a smoothing application with 3 modes: Median, Gaussian,
Each mode has associated params. Median and Gaussian needs a radius,
Anisotropic Filtering needs a time step and a number of iterations.
So below the combobox, the widget displayed depend on the combobox value.
The model we imagined for the parameter list will be a tree more than a
A group of parameters is just one node of the tree, containing a
parameters list, where each parameter can also be a group of parameters.
Please find also some other more specific comments below :
Le 27/04/2011 11:20, Martin Dobias a écrit :
> Hi Camilo
> On Tue, Apr 26, 2011 at 9:46 PM, Camilo Polymeris<cpolymeris at gmail.com> wrote:
>> Hello everyone,
>> I was accepted as a student in GSoC to implement a "SAGA interface for QGIS".
> First of all congratulations to you and to other accepted students for
> the Summer of Code programme!
>> There are also proyects that try to interface QGIS with GRASS and the
>> Orfeo Toolbox (OTB). Paolo has informed me that the creation of a
>> common framework for those systems was discussed and approved during
>> HF. A very interesting idea, IMO, that completely changes the way I
>> would have to work on my project.
> Great that you like the idea of generic framework for processing tools
> in QGIS. Helping us to bring this framework from ideas into code would
> be another great benefit of your project.
>> While I didn't participate in those discussions and don't know what
>> conclusions were drawn, as a brainstorming exercise I am writing a
>> short description of what a very simple API to that framework could
>> look like:
>> Would that more or less fit what you have in mind? Please comment/criticize.
> The discussion was not going into implementation details, we were
> mostly collecting the ideas how the things should work.
> Framework implementation: as you write in the proposal, ideally it
> should be implemented in C++ with Python bindings. I would suggest you
> to start the implementation in Python and skip C++ coding at least at
> the beginning. My experience is that Python is great for this kind of
> prototyping (no compiling, shorter code etc). If you will end up with
> sufficient amount of time at the end of the summer, you can move that
> into C++ and add Python bindings. The idea is not to waste time with
> writing and rewriting various c++ classes when the API is not clear
> yet. Consider this as a suggestion, not a command how to do it :-)
> Forms/Parameters: I would start with with generic input fields (line
> edit / spin box) for integers, floats, strings that can be constrained
> with a range (numbers) or regular expressions (strings) - easily
> achieved with QValidator classes. Then some custom widgets can be
> added for better handling of some types: layer selector, enumeration
> etc. The framework may be extensible so that if a library often uses a
> particular custom type, the plugin integrating the library could add a
> custom widget for it.
Please see 
I fear it will be too limiting to consider parameters only independently
> Advanced parameters: some modules may allow users set lots of
> parameters. Although this is handy, many times the default values work
> well and only few basic parameters are changed regularly. It should be
> possible to mark some parameters as advanced and they would be hidden
> by default, they would show up after clicking a checkbox or a button.
> For example, when creating a vector buffer, the distance would be a
> basic parameter while the number of segments per quadrant would be
> advanced parameter.
Yes basic/advanced parameters seems a must-have to me.
We want not experienced users to use the modules, but we surely want to
give all the possible finetuning possibilities to those who "know what
We would also want checkable parameters.
mandatory parameters should have a value.
non-mandatory parameters would have a checkbox to be activated.
> Categorization: currently we are starting to face some trouble with
> organization of plugins and the "modules" they provide. We thought
> about creating fixed categories as in GRASS (e.g. Raster, Vector,
> Database) where the modules would be placed. We also considered custom
> categories and subcategories. The general problem with categories is
> that many modules do not suit them well if they are not coming from a
> well-organized library such as grass, saga or otb.
> The conclusion that I recall from the discussion at hackfest is that
> tagging is probably the only way to go: each module could have several
> tags assigned to it, then the list of modules would be a tree with top
> level items containing tags and modules would be shown inside these
> tags. Bigger libraries requiring deeper hierarchy might come with
> hierarchical tags like "SAGA/Grid - Tools".
Hierarchical tags seems very reasonable to me.
> The names of tags will be
> left for developers' creativity.
A list of default available tags should be provided.
If the developer feel it can't fit in those, then it can create a new tag.
> This list of processing modules should later get some advanced
> functionality like searching, favourite modules or even attaching most
> used modules on a toolbar, but that is something for the future, we
> are not expecting you to do that during the summer :-) A simple tree
> listing tags and modules assigned to them will suffice.
> Formats and format conversion: for import and export of map layers the
> framework should support any layer loaded (or loadable) by QGIS. In
> case the library uses a different format for input/output, it should
> take care of import/export.
I'm not sure this can be handled in the framework, but more in the
The input images to our modules will be OTB specific data objects, and
the transformation from QGis raster (= filename) to OTB raster would be
done the OTB-QGis plugins implementation (you can check the current
Smoothing application code).
> Running modules from scripts: you probably have that in mind but let's
> keep it explicit: the framework should provide API that would allow
> user to run modules from python console or from a plugin, QGIS would
> basically act just as a default GUI using that API. This will finally
> allow users to run the processing modules in batches - poor man's
> pipelining :-)
> Feedback: many modules are able to return some feedback - either in
> the form of a progress to indicate how long the process will take - or
> to return some messages that will be shown to user. Support for
> cancelling a running module would be helpful too, but not really
> necessary right now.
Must-have also !
Having a common framework to report progress and log messages would be
If its in Qt, it should be through signals.
I already saw some code making a bridge between the OTB events framework
and Qt signals, I'd like to reuse this.
> Qgis-developer mailing list
> Qgis-developer at lists.osgeo.org
More information about the Qgis-developer