[MapProxy] concurrent seed workers on windows

[Brunner Tobias (GEO)]

Hi Oliver,

thanks for the quick reply!

Pillow is on version 3.0.0 ... and as far as I can tell, there is some degree of parallelization, since the performance of mapproxy improves up to around 12 concurrent seed workers, but assigning more seed workers does not improve the performance any further. The WMS requests seem to be issued all at once.

Linux is more of a fall back option, since everything else in our infrastructure is windows based... so I'll probably have to start multiple mapproxy-seeds with different coverages to speed up the seeding process...

Thanks again!
Best
Tobias

-----Ursprüngliche Nachricht-----
Von: Oliver Tonnhofer [mailto:olt at omniscale.de] 
Gesendet: Dienstag, 22. Dezember 2015 10:28
An: Brunner Tobias (GEO) <Tobias.Brunner at zuerich.ch>
Cc: mapproxy at lists.osgeo.org
Betreff: Re: [MapProxy] concurrent seed workers on windows

Hi,


> On 22.12.2015, at 09:14, Brunner Tobias (GEO) <Tobias.Brunner at zuerich.ch> wrote:
> […] 
> I suspect this may have something to do with the python-multiprocessing on windows…?
> Is this a known issue on windows and are there any workarounds?

The way mapproxy-seed is implemented is incompatible with python-multiprocessing on Windows. It therefore uses threading on Windows. Threading in Python is limited in how things can run in parallel (search for GIL if you are interested). More recent Pillow versions are able to "release the GIL” when encoding images, so it should be able to use more than one CPU.

Which version of Pillow/PIL do you use?


> Systems:
> Mapproxy-Windows-VM: 8 cores, 12 GB of ram, windows server 2012 r2, python 2.7 (osgeo4w)
> Mapproxy-linux-VM: 8 cores, 12 gb of ram, redhat

If you have the option: go with linux. 


Regards,
Oliver

-- 
Oliver Tonnhofer  | Omniscale GmbH & Co KG  | http://omniscale.com
OpenStreetMap WMS and tile services         | http://maps.omniscale.com