[MapProxy] Re-enabling mapnik cache per-process while seeding

[Arne Babenhauserheide]

Hi,

Sorry for the strange format of the previous email. I’m now sending it
again in plain format.

------

We are using highly concurrent seeding via Mapnik and a large part of
our processing time is spent re-creating the mapnik.Map(0,0) in
mapproxy/source/mapnik.py

To reduce these costs, I switched the code to one cache per process and
re-enabled the cache. Using 64 cores we saw a speedup of roughly factor
5 to 10, and we would like to contribute these changes back.

The current implementation is a quick hack, but I received permission to
spend a few work hours to clean it up such that it can be included in
mapproxy. Therefore I would like to ask for guidance how to contribute
this change in a way which makes it easy for you to merge it.

The following is the current hack. This hack is far from perfect and has
some side-effects for on-the-fly rendering (not seeding) which I want to
clean up before I create a pull-request, but I hope you can give some
hints how best to move forward to get it ready.


diff --git a/mapproxy/source/mapnik.py b/mapproxy/source/mapnik.py
index 8b87bfc3..69747a48 100644
--- a/mapproxy/source/mapnik.py
+++ b/mapproxy/source/mapnik.py
@@ -18,6 +18,7 @@ from __future__ import absolute_import
 import sys
 import time
 import threading
+import multiprocessing

 from mapproxy.grid import tile_grid
 from mapproxy.image import ImageSource
@@ -26,7 +27,7 @@ from mapproxy.layer import MapExtent,
DefaultMapExtent, BlankImage, MapLayer
 from mapproxy.source import  SourceError
 from mapproxy.client.log import log_request
 from mapproxy.util.py import reraise_exception
-from mapproxy.util.async_ import run_non_blocking
+from mapproxy.util.async import run_non_blocking
 from mapproxy.compat import BytesIO

 try:
@@ -57,8 +58,10 @@ class MapnikSource(MapLayer):
         self.layers = set(layers) if layers else None
         self.scale_factor = scale_factor
         self.lock = lock
-        self._map_objs = {}remove
-        self._map_objs_lock = threading.Lock()
+        global _map_objs
+        _map_objs = {}
+        global _map_objs_lock
+        _map_objs_lock = threading.Lock()
         self._cache_map_obj = reuse_map_objects
         if self.coverage:
             self.extent = MapExtent(self.coverage.bbox, self.coverage.srs)
@@ -94,20 +97,22 @@ class MapnikSource(MapLayer):
             return self.render_mapfile(mapfile, query)

     def map_obj(self, mapfile):
-        if not self._cache_map_obj:
-            m = mapnik.Map(0, 0)
-            mapnik.load_map(m, str(mapfile))
-            return m
-
+        proc = multiprocessing.current_process()
+        process_id = proc._identity # identity is a tuple with a number
+        cachekey = (process_id, mapfile)
         # cache loaded map objects
         # only works when a single proc/thread accesses this object
         # (forking the render process doesn't work because of open database
         #  file handles that gets passed to the child)
-        if mapfile not in self._map_objs:
-            with self._map_objs_lock:
-                if mapfile not in self._map_objs:
+        if cachekey not in _map_objs:
+            with _map_objs_lock:
+                if cachekey not in _map_objs:
                     m = mapnik.Map(0, 0)
+                    _map_objs[cachekey] = m # before load, dangerous!
                     mapnik.load_map(m, str(mapfile))
+
+        return _map_objs[cachekey]

     def render_mapfile(self, mapfile, query):
         return run_non_blocking(self._render_mapfile, (mapfile, query))


Best wishes,
Arne Babenhauserheide

 
-- 
Disy Informationssysteme GmbH 
Arne Babenhauserheide 
Softwareentwickler 
+49 721 16006 443, arne.babenhauserheide at disy.net 
 
Firmensitz: Ludwig-Erhard-Allee 6, 76131 Karlsruhe 
Registergericht: Amtsgericht Mannheim, HRB 107964 
Geschäftsführer: Claus Hofmann 
 
Bitte beachten Sie folgende Informationen für Kunden, Lieferanten und Bewerber 
- Datenschutz: www.disy.net/datenschutz 
- Informationspflichten:  www.disy.net/informationspflichten  
 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: arne_babenhauserheide.vcf
Type: text/x-vcard
Size: 335 bytes
Desc: not available
URL: <http://lists.osgeo.org/pipermail/mapproxy/attachments/20190312/49ba8006/attachment.vcf>