[MapProxy] Performance tuning

[Martin Kokeš]

Hi Attila,

What concerns first case with real machine, was that seeding to mbtiles too? I experienced same problems with mbtiles locking with multiple seeding processes in case of seeding from much slower remote WMSes than any local renderer and definitely given up - now I'm using TileCache directory structure.
 
Martin
  _____  
From: Oliver Tonnhofer [mailto:olt at omniscale.de]
To: BERÉNYI Attila [mailto:aberenyi at gislab.hu]
Cc: mapproxy at lists.osgeo.org
Sent: Mon, 18 Nov 2013 17:57:06 +0100
Subject: Re: [MapProxy] Performance tuning


On 18.11.2013, at 16:54, BERÉNYI Attila wrote:
> I have googled quite a lot to find the information I am looking for - no luck so far. I am well aware this is the hundredth time that someone ask about performance tuning - please bear with me.
> 
> First things first, MapProxy is awesome, well done guys!
> 
> So, I have two separate test cases:
> 	• "Real" machine
> 		• Xeon 5355, 24GB RAM, PostgreSQL 9.3.1-3, PostGIS 2.1.0-2 running on Arch.
> 		• Tile generation chain: PostGIS -> Geoserver -> MapProxy.
> 		• Geographic extent: [-10, 46, 4, 65] (EPSG:4326)
> 		• Zoom levels: 19.
> 		• I have tried several -c (up to 160) parameters (i.e. number of parallel seed processes) but I have never reached the optimal 8-ish re the 15 minutes average load, even though I have adjusted the parameters in Geoserver and PostgreSQL as well.

The source services are the bottleneck in most cases. Have you checked your IO load(e.g. %wa in top)? How many tiles/seconds do you seed? 


> 		• My tests show that even 2 parallel processes per vCPU (i.e. -c 32) cause issue/bottleneck (http://pastebin.com/raw.php?i=29MPZcYp). However the 15 minutes average load is only ~8.

Again, how many tiles/seconds do you seed? SQLite/MBTiles does not work well with heavy read concurrency, but this hasn't been a problem in my seeding processes. Maybe it's necessary to increase the timeout in SQLite during seeding (not configurable ATM).


> And finally a question/suggestion: have you ever considered to utilise the GPU's capabilities within the worker processes (http://gis.stackexchange.com/questions/77181/generate-tiles-using-gpu)?

You should ask on the Mapnik/Geoserver list.


Regards,
Oliver

-- 
Oliver Tonnhofer    | Omniscale GmbH & Co KG    | http://omniscale.com
http://mapproxy.org | https://github.com/olt    | @oltonn







_______________________________________________
MapProxy mailing list
MapProxy at lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/mapproxy