[Pywps-dev] Containerization PyWPS processes

[Jorge Mendes de Jesus]

Hi to all

With new systems you have new problems but new possibilities, another
possibility would be accountabilty, meaning a process (mianly scientific)
can be run, all the logs are inside the docker instance and then commit it
(freeze it)  and  used by another person to check the logs and/or data.

Disk space in docker is a funny thing, can go between 6mb to 600mb in a
blink of an eye by changing OS of not cleaning packages etc etc, so lot of
effort has to be done in optimizing it.  Another advantage is that you can
determine CPU and resources on docker therefore we have a very refined Job
resrouces control

With the new support of Job batch we could extend things to run things in
docker swarms.

A bit from experience.... docker systems need a a bit of "love and
attention" in the beginning and then things run without problems, other
issue is the extremely fast speed of docker development, you prepare things
and the docker community makes some changes and everything  breaks, had
situation once that docker-machine internally was calling some scripts for
package update and during the night someone in the docker community made a
small change and for a couple of hours your couldnt run docker-machine
(until we discovered  it was a big problem).

If this project goes ahead I would ask if Geocat could sponsor it with
working hours.

Cheers
Jorge

On Wed, Sep 20, 2017 at 6:38 PM, Jachym Cepicky <jachym.cepicky at gmail.com>
wrote:

> Hi,
>
> I'm in touch with Adam.
>
>  It has big impact on the disc space, I agree - but afaik, it opens new
> possibilities (imagine, being able to deploy running job to e.g. open shift
> instaces ..) and other system resources impact should not  be that big?
>
> What can I say, all the points raised by Jorge are valid - so let's give
> it a try?
>
> J
>
> st 20. 9. 2017 v 18:07 odesílatel jorge.dejesus <jorge.dejesus at geocat.net>
> napsal:
>
>> Hi to all
>>
>> Interresting research topic, but you have a problem with that approach:
>> starting the process will have a massive overhead (compared with a thread)
>> and will consume alot of disk space and resources  !!!
>>
>> You would have to create the docker image and process  when you install
>> pywps , and then start the docker when the user calls the process.  I am a
>> bit against using such a big system in PyWPS unless someone tries to
>> implement it and run it and show or not that is feasable, *we dont know
>> it until we try it*
>>
>> Those were my 2cent :)
>>
>> J.
>>
>>
>>
>> On 20-09-17 15:41, Adam Laža wrote:
>>
>> Hi devs,
>>
>> I am student of geoinformatics at CTU in Prague. Currently I'm looking
>> for my ma final thesis topic. Yesterday I met with Jachym and we discussed
>> about containerization PyWPS processes (probably with Docker). It could be
>> handy for killing/pausing a process which is as far as I know quite crucial
>> in WPS 2.0.
>>
>> I'd like to know if somebody already researched this posibility or
>> whether you have any suggestion or advice.
>>
>> Thanks in advance.
>> Adam
>>
>>
>> _______________________________________________
>> pywps-dev mailing listpywps-dev at lists.osgeo.orghttps://lists.osgeo.org/mailman/listinfo/pywps-dev
>>
>>
>> _______________________________________________
>> pywps-dev mailing list
>> pywps-dev at lists.osgeo.org
>> https://lists.osgeo.org/mailman/listinfo/pywps-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/pywps-dev/attachments/20170921/d90a4f7c/attachment.html>