[Pywps-dev] maxoperations and asynchronous processes

[Jachym Cepicky]

Hi all,

I would like to talk with you about ticket #65 [1] and commit [2], which
implements this feature.

There are two big changes in the branch. First of all: Process class now
accepts "config" attribute - path to configuration file. In current master,
the configuration file is read *after* the server is initialized - which is
late [3].

Second big change is implementation of maxoperatios and parallelprocesses
configuration options as described in attached image.

General idea is: there are two configuration options: maximum processes
running on the server in parallel and maximum requests stored in queue.

The "queue" is implemented with help of SQLite database, where the incoming
request is stored as serialized JSON string, when there is too many
processes running in paralel.

Every process, once it's finished, it looks into database for processes
stored in queue and if there is some, it takes it and continues with
calculations.

I hope, you like the concept, and that everybody agrees with the proposed
solution: store data into local database. Also number of running processes
and number of stored requests is estimated based on records in database -
is that ok for you?

Beside possibility of storing requests for later execution, this solution
enables us also to kill running processes, based on the their UUID in the
future (and I still hope for pausing).

[image: pywps_maxoperations.png]

Database schema as it is now:

sqlite> .schema

-- table for logging all requests, their status and how did they work
CREATE TABLE pywps_requests(
                uuid VARCHAR(255) not null primary key,
                pid INTEGER not null,
                operation varchar(30) not null,
                version varchar(5) not null,
                time_start text not null,
                time_end text,
                identifier text,
                message text,
                percent_done float,
                status varchar(30)
            );

-- table for saving and restoring requets in to queue
CREATE TABLE pywps_stored_requests(
                uuid VARCHAR(255) not null primary key,
                request BLOB not null
            );

Anybody want's to test this? Little code review would not harm.

Any objections to put this to master?

Thanks

Jachym

[1] https://github.com/geopython/pywps/issues/65
[2]
https://github.com/geopython/pywps/commit/7c6875995d24fde77d60e8b5014c52c62bcc632e
[3]
https://github.com/geopython/pywps/commit/7c6875995d24fde77d60e8b5014c52c62bcc632e#diff-1ab67f2df437e13a3ace4c4c031e1a68R33
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/pywps-dev/attachments/20160422/cfa6c7f7/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: pywps_maxoperations.png
Type: image/png
Size: 78108 bytes
Desc: not available
URL: <http://lists.osgeo.org/pipermail/pywps-dev/attachments/20160422/cfa6c7f7/attachment-0001.png>