[MapProxy] Performance tuning

[BERÉNYI Attila]

Gents,

Thanks very much for the replies.

Oliver,

I have not checked the I/O rate, but I will definitely do. I am seeding to
SQLite from PostGIS, but I guess if there are issues with heavy read
concurrency it could be the same for heavy write concurrency.

Martin,

The source is a PostGIS running on the same machine so there is no network
overhead or remote sources, but I'll give the file cache a go, see what
comes out.

Cheers,
Attila


2013/11/19 Martin Kokeš <shr3k at typo3-hosting.com>

> Hi Attila,
>
> What concerns first case with real machine, was that seeding to mbtiles
> too? I experienced same problems with mbtiles locking with multiple seeding
> processes in case of seeding from much slower remote WMSes than any local
> renderer and definitely given up - now I'm using TileCache directory
> structure.
>
> Martin
>   _____
> From: Oliver Tonnhofer [mailto:olt at omniscale.de]
> To: BERÉNYI Attila [mailto:aberenyi at gislab.hu]
> Cc: mapproxy at lists.osgeo.org
> Sent: Mon, 18 Nov 2013 17:57:06 +0100
> Subject: Re: [MapProxy] Performance tuning
>
>
> On 18.11.2013, at 16:54, BERÉNYI Attila wrote:
> > I have googled quite a lot to find the information I am looking for - no
> luck so far. I am well aware this is the hundredth time that someone ask
> about performance tuning - please bear with me.
> >
> > First things first, MapProxy is awesome, well done guys!
> >
> > So, I have two separate test cases:
> >       * "Real" machine
> >               * Xeon 5355, 24GB RAM, PostgreSQL 9.3.1-3, PostGIS 2.1.0-2
> running on Arch.
> >               * Tile generation chain: PostGIS -> Geoserver -> MapProxy.
> >               * Geographic extent: [-10, 46, 4, 65] (EPSG:4326)
> >               * Zoom levels: 19.
> >               * I have tried several -c (up to 160) parameters (i.e.
> number of parallel seed processes) but I have never reached the optimal
> 8-ish re the 15 minutes average load, even though I have adjusted the
> parameters in Geoserver and PostgreSQL as well.
>
> The source services are the bottleneck in most cases. Have you checked
> your IO load(e.g. %wa in top)? How many tiles/seconds do you seed?
>
>
> >               * My tests show that even 2 parallel processes per vCPU
> (i.e. -c 32) cause issue/bottleneck (
> http://pastebin.com/raw.php?i=29MPZcYp). However the 15 minutes average
> load is only ~8.
>
> Again, how many tiles/seconds do you seed? SQLite/MBTiles does not work
> well with heavy read concurrency, but this hasn't been a problem in my
> seeding processes. Maybe it's necessary to increase the timeout in SQLite
> during seeding (not configurable ATM).
>
>
> > And finally a question/suggestion: have you ever considered to utilise
> the GPU's capabilities within the worker processes (
> http://gis.stackexchange.com/questions/77181/generate-tiles-using-gpu)?
>
> You should ask on the Mapnik/Geoserver list.
>
>
> Regards,
> Oliver
>
> --
> Oliver Tonnhofer    | Omniscale GmbH & Co KG    | http://omniscale.com
> http://mapproxy.org | https://github.com/olt    | @oltonn
>
>
>
>
>
>
>
> _______________________________________________
> MapProxy mailing list
> MapProxy at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/mapproxy
> _______________________________________________
> MapProxy mailing list
> MapProxy at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/mapproxy
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/mapproxy/attachments/20131120/fa7e3343/attachment.html>