[MapProxy] Threading & Memory Errors
Keith Moss
Keith.Moss at landgate.wa.gov.au
Sun Feb 24 19:38:03 PST 2013
Hi list,
We've started noticing memory and threading errors. (stacktrace attached below)
We've noticed the "can't start new thread" errors in small amounts previously, but in the past couple of days (which did correspond roughly with spikes in usage) we've seen them in much larger numbers (~2000 on Sunday, another 600 this morning before Apache restarted itself). The "out of memory" errors have different stacktraces each time (as you might expect). I haven't got any data yet on what memoy usage on the server is at these times but I've never seen it come close to maxing out the available memory when I've been monitoring it. The httpd process does seem to be gradually growing its memory usage all of the time, though.
I've only got second hand experience of what's occurring, but I'm told initially it was only a handful of tile requests failing and throuought the day it grew gradually over a 12-or-so hour period until every request was returning an error.
It "corrected" itself when Apache hit its MaxRequestsPerChild and restarted the httpd process. For the time being I've reduced our MaxRequestsPerChild from 60k to 30k to see if that can work around the issue.
Over the past few days we've had calls to MapProxy as follows:
20/02/2013: 64k
21/02/2013: 104k
22/02/2013: 64k
23/02/2013: 33k
24/02/2013: 33k (when we started seeing the "can't start thread" errors)
25/02/2013: 20k (thus far today)
For reference: MapProxy 1.5.0 running in Apache 2. 2 with mod_wsgi on Windows Server 2008.
[Mon Feb 25 00:00:15 2013] [error] [client 202.1.16.73] Traceback (most recent call last):
[Mon Feb 25 00:00:15 2013] [error] [client 202.1.16.73] File "c:\\pythonenv\\mapproxy-srss\\Lib\\site-packages\\mapproxy-1.5.0a_20130221-py2.7.egg\\mapproxy\\wsgiapp.py", line 166, in __call__
[Mon Feb 25 00:00:15 2013] [error] [client 202.1.16.73] resp = self.handlers[handler_name].handle(req)
[Mon Feb 25 00:00:15 2013] [error] [client 202.1.16.73] File "c:\\pythonenv\\mapproxy-srss\\Lib\\site-packages\\mapproxy-1.5.0a_20130221-py2.7.egg\\mapproxy\\service\\ows.py", line 38, in handle
[Mon Feb 25 00:00:15 2013] [error] [client 202.1.16.73] return self.services[service].handle(req)
[Mon Feb 25 00:00:15 2013] [error] [client 202.1.16.73] File "c:\\pythonenv\\mapproxy-srss\\Lib\\site-packages\\mapproxy-1.5.0a_20130221-py2.7.egg\\mapproxy\\service\\base.py", line 30, in handle
[Mon Feb 25 00:00:15 2013] [error] [client 202.1.16.73] return handler(parsed_req)
[Mon Feb 25 00:00:15 2013] [error] [client 202.1.16.73] File "c:\\pythonenv\\mapproxy-srss\\Lib\\site-packages\\mapproxy-1.5.0a_20130221-py2.7.egg\\mapproxy\\service\\wms.py", line 111, in map
[Mon Feb 25 00:00:15 2013] [error] [client 202.1.16.73] renderer.render(merger)
[Mon Feb 25 00:00:15 2013] [error] [client 202.1.16.73] File "c:\\pythonenv\\mapproxy-srss\\Lib\\site-packages\\mapproxy-1.5.0a_20130221-py2.7.egg\\mapproxy\\service\\wms.py", line 471, in render
[Mon Feb 25 00:00:15 2013] [error] [client 202.1.16.73] layer_merger)
[Mon Feb 25 00:00:15 2013] [error] [client 202.1.16.73] File "c:\\pythonenv\\mapproxy-srss\\Lib\\site-packages\\mapproxy-1.5.0a_20130221-py2.7.egg\\mapproxy\\util\\async.py", line 204, in _single_call
[Mon Feb 25 00:00:15 2013] [error] [client 202.1.16.73] result = func(*args)
[Mon Feb 25 00:00:15 2013] [error] [client 202.1.16.73] File "c:\\pythonenv\\mapproxy-srss\\Lib\\site-packages\\mapproxy-1.5.0a_20130221-py2.7.egg\\mapproxy\\service\\wms.py", line 513, in _render_layer
[Mon Feb 25 00:00:15 2013] [error] [client 202.1.16.73] layer_img = layer.get_map(self.query)
[Mon Feb 25 00:00:15 2013] [error] [client 202.1.16.73] File "c:\\pythonenv\\mapproxy-srss\\Lib\\site-packages\\mapproxy-1.5.0a_20130221-py2.7.egg\\mapproxy\\layer.py", line 327, in get_map
[Mon Feb 25 00:00:15 2013] [error] [client 202.1.16.73] return layer.get_map(query)
[Mon Feb 25 00:00:15 2013] [error] [client 202.1.16.73] File "c:\\pythonenv\\mapproxy-srss\\Lib\\site-packages\\mapproxy-1.5.0a_20130221-py2.7.egg\\mapproxy\\layer.py", line 404, in get_map
[Mon Feb 25 00:00:15 2013] [error] [client 202.1.16.73] result = self._image(query)
[Mon Feb 25 00:00:15 2013] [error] [client 202.1.16.73] File "c:\\pythonenv\\mapproxy-srss\\Lib\\site-packages\\mapproxy-1.5.0a_20130221-py2.7.egg\\mapproxy\\layer.py", line 437, in _image
[Mon Feb 25 00:00:15 2013] [error] [client 202.1.16.73] tile_collection = self.tile_manager.load_tile_coords(affected_tile_coords, with_metadata=query.tiled_only)
[Mon Feb 25 00:00:15 2013] [error] [client 202.1.16.73] File "c:\\pythonenv\\mapproxy-srss\\Lib\\site-packages\\mapproxy-1.5.0a_20130221-py2.7.egg\\mapproxy\\cache\\tile.py", line 131, in load_tile_coords
[Mon Feb 25 00:00:15 2013] [error] [client 202.1.16.73] created_tiles = creator.create_tiles(uncached_tiles)
[Mon Feb 25 00:00:15 2013] [error] [client 202.1.16.73] File "c:\\pythonenv\\mapproxy-srss\\Lib\\site-packages\\mapproxy-1.5.0a_20130221-py2.7.egg\\mapproxy\\cache\\tile.py", line 260, in create_tiles
[Mon Feb 25 00:00:15 2013] [error] [client 202.1.16.73] created_tiles = self._create_meta_tiles(meta_tiles)
[Mon Feb 25 00:00:15 2013] [error] [client 202.1.16.73] File "c:\\pythonenv\\mapproxy-srss\\Lib\\site-packages\\mapproxy-1.5.0a_20130221-py2.7.egg\\mapproxy\\cache\\tile.py", line 329, in _create_meta_tiles
[Mon Feb 25 00:00:15 2013] [error] [client 202.1.16.73] return self._create_threaded(self._create_meta_tile, meta_tiles)
[Mon Feb 25 00:00:15 2013] [error] [client 202.1.16.73] File "c:\\pythonenv\\mapproxy-srss\\Lib\\site-packages\\mapproxy-1.5.0a_20130221-py2.7.egg\\mapproxy\\cache\\tile.py", line 276, in _create_threaded
[Mon Feb 25 00:00:15 2013] [error] [client 202.1.16.73] for new_tiles in async_pool.imap(create_func, tiles):
[Mon Feb 25 00:00:15 2013] [error] [client 202.1.16.73] File "c:\\pythonenv\\mapproxy-srss\\Lib\\site-packages\\mapproxy-1.5.0a_20130221-py2.7.egg\\mapproxy\\util\\async.py", line 54, in _result_iter
[Mon Feb 25 00:00:15 2013] [error] [client 202.1.16.73] for result in results:
[Mon Feb 25 00:00:15 2013] [error] [client 202.1.16.73] File "c:\\pythonenv\\mapproxy-srss\\Lib\\site-packages\\mapproxy-1.5.0a_20130221-py2.7.egg\\mapproxy\\util\\async.py", line 182, in map_each
[Mon Feb 25 00:00:15 2013] [error] [client 202.1.16.73] self.pool = self._init_pool()
[Mon Feb 25 00:00:15 2013] [error] [client 202.1.16.73] File "c:\\pythonenv\\mapproxy-srss\\Lib\\site-packages\\mapproxy-1.5.0a_20130221-py2.7.egg\\mapproxy\\util\\async.py", line 274, in _init_pool
[Mon Feb 25 00:00:15 2013] [error] [client 202.1.16.73] t.start()
[Mon Feb 25 00:00:15 2013] [error] [client 202.1.16.73] File "\\\\10.0.0.69\\c_drive\\Python27\\Lib\\threading.py", line 497, in start
[Mon Feb 25 00:00:15 2013] [error] [client 202.1.16.73] _start_new_thread(self.__bootstrap, ())
[Mon Feb 25 00:00:15 2013] [error] [client 202.1.16.73] error: can't start new thread
__________________________________
Keith Moss
Satellite Remote Sensing Services
Operations
Landgate
65 Brockway Road, FLOREAT WA 6014
PO Box 741 Wembley WA 6913
Ph. (08) 9387 0332 | Fx. (08) 9383 7142
keith.moss at landgate.wa.gov.au<mailto:keith.moss at landgate.wa.gov.au>
http://www.landgate.wa.gov.au/
This e-mail and any files transmitted with it are intended only for the use of the addressee(s). It may contain information that is confidential and privileged, in which case neither is intended to be waived or lost by mistaken delivery to you. If you are not an intended recipient, any use, interference with, disclosure, distribution or copying of this material is unauthorised and prohibited. If you receive this e-mail in error, please notify the sender by return e-mail and delete the message and any attachments from your system. Unless specifically indicated, this e-mail does not constitute formal advice or commitment by the sender or the Western Australian Land Information Authority (Landgate). Information in this message not relating to the official business of Landgate shall be understood as neither given nor endorsed by it. It is your responsibility to check any attachments for viruses and defects before opening or sending them on. Landgate’s liability is limited to re-supplying affected attachments.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/mapproxy/attachments/20130225/de2f8347/attachment.html>
More information about the MapProxy
mailing list