[Pywps-dev] scheduler extension for pywps

[Carsten Ehbrecht]

Hi *,

in our copernicus project we want to provide processing functionality
using PyWPS next to a climate data store. To handle the workload we need
to attach batch job scheduler systems like slurm
(https://slurm.schedmd.com/), grid-engine and torque. We are also
interested in using docker/container to run WPS processes ... but our
compute-providers are not ready for it yet.

I've prepared a demo to show how such a PyWPS/extension might look like.
When this extension is ready we would like to contribute it to PyWPS.

Would be nice to get some comments from you :)

I've added a "processing" package to PyWPS which by default uses
multiprocessing but can also delegate processing jobs to a scheduler:

https://github.com/bird-house/pywps/tree/dev-scheduler/pywps/processing

It uses the drmaa python library to talk to a scheduling service:
https://pypi.python.org/pypi/drmaa

You can try the docker demo with slurm:
https://github.com/bird-house/birdhouse-playground/tree/master/docker/pywps-scheduler-demo

Maybe the "processing" interface could also be used as an
extension-point for processing with docker/containers.

Cheers,
Carsten
-- 
Carsten Ehbrecht
Abteilung Datenmanagement

Deutsches Klimarechenzentrum GmbH (DKRZ)
Bundesstraße 45 a • D-20146 Hamburg • Germany

Phone: +49 40 460094-148
FAX:   +49 40 460094-270
Email: ehbrecht at dkrz.de
URL:   www.dkrz.de

Geschäftsführer: Prof. Dr. Thomas Ludwig
Sitz der Gesellschaft: Hamburg
Amtsgericht Hamburg HRB 39784