[Pywps-dev] scheduler extension for pywps

[Carsten Ehbrecht]

Hi Jachym,

i've made a pull request:
https://github.com/geopython/pywps/pull/271

I've added an "extensions" section to the documentation describing this
scheduler extension. A diagram shows the interactions between PyWPS and
the scheduler system.

It should be save to merge ... you need to decide if the
"pywps.processing" interface fits to your demands. You probably need to
adapt and change it when using celery.

Cheers,
Carsten

On 06/04/2017 10:30 PM, Jachym Cepicky wrote:
> Carsten,
> 
> this is just cool! Could you add description and some user documentation
> to docs? Can we put this into `development` branch of PyWPS?
> 
> Thanks a lot!
> 
> Jachym
> 
> po 29. 5. 2017 v 13:46 odesílatel Carsten Ehbrecht <ehbrecht at dkrz.de
> <mailto:ehbrecht at dkrz.de>> napsal:
> 
>     Hi *,
> 
>     in our copernicus project we want to provide processing functionality
>     using PyWPS next to a climate data store. To handle the workload we need
>     to attach batch job scheduler systems like slurm
>     (https://slurm.schedmd.com/), grid-engine and torque. We are also
>     interested in using docker/container to run WPS processes ... but our
>     compute-providers are not ready for it yet.
> 
>     I've prepared a demo to show how such a PyWPS/extension might look like.
>     When this extension is ready we would like to contribute it to PyWPS.
> 
>     Would be nice to get some comments from you :)
> 
>     I've added a "processing" package to PyWPS which by default uses
>     multiprocessing but can also delegate processing jobs to a scheduler:
> 
>     https://github.com/bird-house/pywps/tree/dev-scheduler/pywps/processing
> 
>     It uses the drmaa python library to talk to a scheduling service:
>     https://pypi.python.org/pypi/drmaa
> 
>     You can try the docker demo with slurm:
>     https://github.com/bird-house/birdhouse-playground/tree/master/docker/pywps-scheduler-demo
> 
>     Maybe the "processing" interface could also be used as an
>     extension-point for processing with docker/containers.
> 
>     Cheers,
>     Carsten
>     --
>     Carsten Ehbrecht
>     Abteilung Datenmanagement
> 
>     Deutsches Klimarechenzentrum GmbH (DKRZ)
>     Bundesstraße 45 a • D-20146 Hamburg • Germany
> 
>     Phone: +49 40 460094-148 <tel:+49%2040%20460094148>
>     FAX:   +49 40 460094-270 <tel:+49%2040%20460094270>
>     Email: ehbrecht at dkrz.de <mailto:ehbrecht at dkrz.de>
>     URL:   www.dkrz.de <http://www.dkrz.de>
> 
>     Geschäftsführer: Prof. Dr. Thomas Ludwig
>     Sitz der Gesellschaft: Hamburg
>     Amtsgericht Hamburg HRB 39784
>     _______________________________________________
>     pywps-dev mailing list
>     pywps-dev at lists.osgeo.org <mailto:pywps-dev at lists.osgeo.org>
>     https://lists.osgeo.org/mailman/listinfo/pywps-dev
> 

-- 
Carsten Ehbrecht
Abteilung Datenmanagement

Deutsches Klimarechenzentrum GmbH (DKRZ)
Bundesstraße 45 a • D-20146 Hamburg • Germany

Phone: +49 40 460094-148
FAX:   +49 40 460094-270
Email: ehbrecht at dkrz.de
URL:   www.dkrz.de

Geschäftsführer: Prof. Dr. Thomas Ludwig
Sitz der Gesellschaft: Hamburg
Amtsgericht Hamburg HRB 39784