[Pywps-dev] Docker extension

[Jorge Mendes de Jesus]

Hi Adam

The idea of multiprocessing is to we run things in multiple cores out of
the GIL, if your process is running in a docker container it should be run
in what ever core it is available. Therefore having a Python GIL thread
using checking processes that run in different systems make sense, since
you experience problems (IMHO).

For the record, in another project I had bad experience using multiprocess
on python3.5 on ubuntu server, then I put all the mutiprocessing code in a
docker container running alpine in  docker-machine running KVM and
everyything worked 100% . It maybe that multiprocess lib has some problems

Jorge

On Thu, Mar 22, 2018 at 4:51 PM, Adam Laža <ad.laza32 at gmail.com> wrote:

> Hi Jorge,
>
> original idea was to use multiprocessing package for the deamon that
> manages containers removal. However zombie processes accumulated on the
> server when using multiprocessing. Using threads solved the problem - no
> zombie processes remain on the server. Nevertheless I wanted to point on it
> to know if it's acceptable solution for you.
>
> Also before the PR, I would appreciate to get rid of dependency on my fork
> of OWSLib where I use async/sync switch. We already discussed it at ML[1]
> with Carsten. Can you guys now when you're all together merge Carsten's
> solution into master of OWSLib?
>
> Cheers,
> A.
>
> [1] https://lists.osgeo.org/pipermail/owslib-devel/2018-
> January/000304.html
>
> 2018-03-22 16:22 GMT+01:00 Jorge Mendes de Jesus <jorge.dejesus at geocat.net
> >:
>
>> Hi Adam
>> Stilll didnt had the problem  to look what you did on on your fork,
>>
>> We added your email to the issue concerning containers, so that we keep
>> track
>> https://github.com/geopython/pywps/issues/61
>>
>> If the problems with threads are solved when you can request a PR and we
>> will merge it
>>
>> Cheers
>> Jorge
>>
>>
>> On Mon, Mar 12, 2018 at 8:58 PM, Adam Laža <ad.laza32 at gmail.com> wrote:
>>
>>> Hi devs,
>>>
>>> during work on my thesis[1] I've added Docker extension to pywps. At my
>>> git there is a demo server (pywps-demo with extended config file) [2]. To
>>> run it with Docker extension it's necessary to use pywps[3] and OWSLib[4]
>>> from my git. Also you need Docker engine to build image and create
>>> container (the extension was tested only on Linux).
>>>
>>> I will be glad for any comment. Recently, I faced a problem with zombie
>>> processes when using multiprocessing python package for deamon that manages
>>> created docker containers. Eventually, I used threading[5] instead but not
>>> really sure if it's the right solution. So any opinion will be appreciated.
>>>
>>> Cheers,
>>> Adam
>>>
>>> [1] https://github.com/ctu-geoforall-lab-projects/dp-laza-2018/b
>>> lob/master/text/adam-laza-dp-2018.pdf
>>> [2] https://github.com/lazaa32/pywps-flask
>>> [3] https://github.com/lazaa32/pywps
>>> [4] https://github.com/lazaa32/OWSLib/tree/sync_request
>>> [5] https://github.com/lazaa32/pywps/blob/develop/pywps/processi
>>> ng/container.py#L73
>>>
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/pywps-dev/attachments/20180325/3ec016b3/attachment.html>