[Qgis-developer] Running QGIS and plugins on cluster

[Alex Mandel]

On 04/14/2015 06:43 AM, Alexander Bruy wrote:
> Hi,
> 
> it is not a secret that running analysis tasks on a large datasets
> may be time consuming. One of the possible solutions if to run
> analysis in parallel.
> 
> So I'm wondering is it possible and makes sense to run QGIS plugin
> on cluster to speed-up analysis. For example using MPI (through pypar
> or mpi4py)? Anyone has such experience?
> 

No reason you can't do something with MPI. Though that is a more
complicated cluster setup.

Plenty of work has been done in the past on using GRASS in a cluster
http://grasswiki.osgeo.org/wiki/Parallel_GRASS_jobs

If you have access to a cluster then the answer might depend on what
kind of compute cluster. I would actually try to abstract it behind
something like a WPS server and have QGIS just talk to the WPS server
that then uses GRASS, SAGA, Python(Shapely etc) or whatever cluster
computations you want in the background.

Also consider what you might actually want on a single compute node is
simply local parallel processing using the built in python multi core
support tools. Keep in mind that splitting jobs in a map/reduce way only
works for some types of computation.

Thanks,
Alex