[Pywps-dev] scheduler extension for pywps

[Jachym Cepicky]

Carsten,

this is just cool! Could you add description and some user documentation to
docs? Can we put this into `development` branch of PyWPS?

Thanks a lot!

Jachym

po 29. 5. 2017 v 13:46 odesílatel Carsten Ehbrecht <ehbrecht at dkrz.de>
napsal:

> Hi *,
>
> in our copernicus project we want to provide processing functionality
> using PyWPS next to a climate data store. To handle the workload we need
> to attach batch job scheduler systems like slurm
> (https://slurm.schedmd.com/), grid-engine and torque. We are also
> interested in using docker/container to run WPS processes ... but our
> compute-providers are not ready for it yet.
>
> I've prepared a demo to show how such a PyWPS/extension might look like.
> When this extension is ready we would like to contribute it to PyWPS.
>
> Would be nice to get some comments from you :)
>
> I've added a "processing" package to PyWPS which by default uses
> multiprocessing but can also delegate processing jobs to a scheduler:
>
> https://github.com/bird-house/pywps/tree/dev-scheduler/pywps/processing
>
> It uses the drmaa python library to talk to a scheduling service:
> https://pypi.python.org/pypi/drmaa
>
> You can try the docker demo with slurm:
>
> https://github.com/bird-house/birdhouse-playground/tree/master/docker/pywps-scheduler-demo
>
> Maybe the "processing" interface could also be used as an
> extension-point for processing with docker/containers.
>
> Cheers,
> Carsten
> --
> Carsten Ehbrecht
> Abteilung Datenmanagement
>
> Deutsches Klimarechenzentrum GmbH (DKRZ)
> Bundesstraße 45 a • D-20146 Hamburg • Germany
>
> Phone: +49 40 460094-148 <+49%2040%20460094148>
> FAX:   +49 40 460094-270 <+49%2040%20460094270>
> Email: ehbrecht at dkrz.de
> URL:   www.dkrz.de
>
> Geschäftsführer: Prof. Dr. Thomas Ludwig
> Sitz der Gesellschaft: Hamburg
> Amtsgericht Hamburg HRB 39784
> _______________________________________________
> pywps-dev mailing list
> pywps-dev at lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/pywps-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/pywps-dev/attachments/20170604/7e39e0eb/attachment.html>