<div>Before we turn on debugging mode let's try a couple of simple changes to your script:</div><div><br></div><div><a href="http://pastebin.ca/2165984">http://pastebin.ca/2165984</a></div><div><br></div><div>I have replaced the call to msCleanup with a call to msIO_resetHandlers which clears all buffers.</div>
<div><br></div><div><a href="http://pastebin.ca/2165986">http://pastebin.ca/2165986</a></div><div><br></div><div>same as before but with the buffer accessed as a string rather than a byte array (string is not copied inside mapscript, array is).</div>
<div><br></div><div>Let us now how this goes.</div><div><br></div><div>Umberto<br><br><div class="gmail_quote">On Thu, Jun 28, 2012 at 8:02 PM, Stephan Meißl <span dir="ltr"><<a href="mailto:stephan@meissl.name" target="_blank">stephan@meissl.name</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Thomas, Umberto,<br>
<br>
we created a simple (at least as simple as possible) test case. Here is<br>
a basic Python script: <a href="http://pastebin.ca/2165345" target="_blank">http://pastebin.ca/2165345</a> which we deployed in<br>
Apache using mod_wsgi with this config: <a href="http://pastebin.ca/2165346" target="_blank">http://pastebin.ca/2165346</a><br>
<br>
There is also a basic mapfile using the "road" data in "msautotets/wxs"<br>
<a href="http://pastebin.ca/2165329" target="_blank">http://pastebin.ca/2165329</a> but I suspect you can use whatever mapfile<br>
you have at hand.<br>
<br>
When we issue multiple simultaneous requests e.g. using this script<br>
<a href="http://pastebin.ca/2165348" target="_blank">http://pastebin.ca/2165348</a> (note the "&" at the end of the curl call)<br>
some of the requests result in errors.<br>
<br>
Either removing the call to msCleanup() or extending the Apache config<br>
to limit the number of threads to 1 (<a href="http://pastebin.ca/2165347" target="_blank">http://pastebin.ca/2165347</a>) seems<br>
to resolve the issues but of course with limitations.<br>
<br>
Are we missing anything essential? How should we free request/map<br>
objects without using msCleanup() in Python?<br>
<br>
Thanks for any hint,<br>
Stephan<br>
<div class="HOEnZb"><div class="h5"><br>
<br>
On Thu, 2012-06-28 at 14:15 +0200, Umberto Nicoletti wrote:<br>
> Fabian,<br>
> if you could put together a test case that I can run to reproduce the<br>
> problem I will take a look into it.<br>
><br>
><br>
> Regards,<br>
> Umberto<br>
><br>
> On Thu, Jun 28, 2012 at 1:44 PM, thomas bonfort<br>
> <<a href="mailto:thomas.bonfort@gmail.com">thomas.bonfort@gmail.com</a>> wrote:<br>
> Fabian,<br>
> can you get a backtrace of the segfaults?<br>
><br>
> Aside from that, I would personally refrain from using threads<br>
> with<br>
> mapscript and stick to a prefork mpm in apache to accomplish<br>
> this. It<br>
> is also my understanding that calling msCleanup() should be<br>
> limited to<br>
> when a process is actually exiting, and not at the end of each<br>
> request, i.e. if I were you I would investigate where memory<br>
> is<br>
> leaking when freeing your request/map objects.<br>
><br>
> regards,<br>
> thomas<br>
><br>
> On Thu, Jun 28, 2012 at 11:28 AM, Fabian Schindler<br>
> <<a href="mailto:fabian.schindler@eox.at">fabian.schindler@eox.at</a>> wrote:<br>
> > Hi all,<br>
> ><br>
> > Has anyone had issues with Python MapScript and the<br>
> msCleanup() in<br>
> > WSGI-Apache applications? Using msCleanup() results in<br>
> strange<br>
> > errors/segfaults in multi-threaded setups but not using it<br>
> quickly eats up<br>
> > memory.<br>
> ><br>
> > We are developing a Django application to serve enhanced WCS<br>
> responses<br>
> > (according to the EO-WCS extension). For this purpose, we<br>
> use MapScript to<br>
> > receive a basic WCS response, which we are extending with<br>
> additional<br>
> > metadata.<br>
> ><br>
> > Our mapserver dispatch code is like this:<br>
> ><br>
> > mapscript.msIO_installStdoutToBuffer()<br>
> > dispatch_status = map.OWSDispatch(ows_req)<br>
> > content_type = mapscript.msIO_stripStdoutBufferContentType()<br>
> > mapscript.msIO_stripStdoutBufferContentHeaders()<br>
> > result = mapscript.msIO_getStdoutBufferBytes()<br>
> > mapscript.msCleanup(1)<br>
> > return result, content_type, dispatch_status<br>
> ><br>
> > Unfortunately we experienced several weird errors<br>
> (segfaults, hit assertions<br>
> > within GEOS, etc) when dispatching requests, especially in<br>
> multi-threaded<br>
> > environments (such as a deployment with Apache). We traced<br>
> this down and<br>
> > found that a call to `msCleanup` right after the OWSDispatch<br>
> was causing the<br>
> > problems.<br>
> ><br>
> > Taken from the mapscript documentation:<br>
> ><br>
> > "msCleanup() attempts to recover all dynamically allocated<br>
> resources<br>
> > allocated by MapServer code and dependent libraries. It it<br>
> used primarily<br>
> > for final cleanup in scripts that need to do memory leak<br>
> testing to get rid<br>
> > of “noise” one-time allocations. It should not normally be<br>
> used by<br>
> > production code."<br>
> ><br>
> > When we remove the call to msCleanup, no segfaults and other<br>
> errors occur<br>
> > within Apache. Unfortunately, while running our test cases<br>
> (including<br>
> > several hundreds MapServer dispatches) we quickly run out of<br>
> memory (e.g:<br>
> > memory consumption after 30 test cases was over 2.2 GiB<br>
> instead of about 150<br>
> > MB).<br>
> ><br>
> > Any ideas are welcome :)<br>
> ><br>
> > Thanks,<br>
> > Fabian<br>
> ><br>
> > _______________________________________________<br>
> > mapserver-dev mailing list<br>
> > <a href="mailto:mapserver-dev@lists.osgeo.org">mapserver-dev@lists.osgeo.org</a><br>
> > <a href="http://lists.osgeo.org/mailman/listinfo/mapserver-dev" target="_blank">http://lists.osgeo.org/mailman/listinfo/mapserver-dev</a><br>
> _______________________________________________<br>
> mapserver-dev mailing list<br>
> <a href="mailto:mapserver-dev@lists.osgeo.org">mapserver-dev@lists.osgeo.org</a><br>
> <a href="http://lists.osgeo.org/mailman/listinfo/mapserver-dev" target="_blank">http://lists.osgeo.org/mailman/listinfo/mapserver-dev</a><br>
><br>
><br>
><br>
> _______________________________________________<br>
> mapserver-dev mailing list<br>
> <a href="mailto:mapserver-dev@lists.osgeo.org">mapserver-dev@lists.osgeo.org</a><br>
> <a href="http://lists.osgeo.org/mailman/listinfo/mapserver-dev" target="_blank">http://lists.osgeo.org/mailman/listinfo/mapserver-dev</a><br>
<br>
<br>
</div></div></blockquote></div><br></div>