[mapserver-dev] python mapscript and msCleanup issues

[Umberto Nicoletti]

Before we turn on debugging mode let's try a couple of simple changes to
your script:

http://pastebin.ca/2165984

I have replaced the call to msCleanup with a call to msIO_resetHandlers
which clears all buffers.

http://pastebin.ca/2165986

same as before but with the buffer accessed as a string rather than a byte
array (string is not copied inside mapscript, array is).

Let us now how this goes.

Umberto

On Thu, Jun 28, 2012 at 8:02 PM, Stephan Meißl <stephan at meissl.name> wrote:

> Thomas, Umberto,
>
> we created a simple (at least as simple as possible) test case. Here is
> a basic Python script: http://pastebin.ca/2165345 which we deployed in
> Apache using mod_wsgi with this config: http://pastebin.ca/2165346
>
> There is also a basic mapfile using the "road" data in "msautotets/wxs"
> http://pastebin.ca/2165329 but I suspect you can use whatever mapfile
> you have at hand.
>
> When we issue multiple simultaneous requests e.g. using this script
> http://pastebin.ca/2165348 (note the "&" at the end of the curl call)
> some of the requests result in errors.
>
> Either removing the call to msCleanup() or extending the Apache config
> to limit the number of threads to 1 (http://pastebin.ca/2165347) seems
> to resolve the issues but of course with limitations.
>
> Are we missing anything essential? How should we free request/map
> objects without using msCleanup() in Python?
>
> Thanks for any hint,
> Stephan
>
>
> On Thu, 2012-06-28 at 14:15 +0200, Umberto Nicoletti wrote:
> > Fabian,
> > if you could put together a test case that I can run to reproduce the
> > problem I will take a look into it.
> >
> >
> > Regards,
> > Umberto
> >
> > On Thu, Jun 28, 2012 at 1:44 PM, thomas bonfort
> > <thomas.bonfort at gmail.com> wrote:
> >         Fabian,
> >         can you get a backtrace of the segfaults?
> >
> >         Aside from that, I would personally refrain from using threads
> >         with
> >         mapscript and stick to a prefork mpm in apache to accomplish
> >         this. It
> >         is also my understanding that calling msCleanup() should be
> >         limited to
> >         when a process is actually exiting, and not at the end of each
> >         request, i.e. if I were you I would investigate where memory
> >         is
> >         leaking when freeing your request/map objects.
> >
> >         regards,
> >         thomas
> >
> >         On Thu, Jun 28, 2012 at 11:28 AM, Fabian Schindler
> >         <fabian.schindler at eox.at> wrote:
> >         > Hi all,
> >         >
> >         > Has anyone had issues with Python MapScript and the
> >         msCleanup() in
> >         > WSGI-Apache applications? Using msCleanup() results in
> >         strange
> >         > errors/segfaults in multi-threaded setups but not using it
> >         quickly eats up
> >         > memory.
> >         >
> >         > We are developing a Django application to serve enhanced WCS
> >         responses
> >         > (according to the EO-WCS extension). For this purpose, we
> >         use MapScript to
> >         > receive a basic WCS response, which we are extending with
> >         additional
> >         > metadata.
> >         >
> >         > Our mapserver dispatch code is like this:
> >         >
> >         > mapscript.msIO_installStdoutToBuffer()
> >         > dispatch_status = map.OWSDispatch(ows_req)
> >         > content_type = mapscript.msIO_stripStdoutBufferContentType()
> >         > mapscript.msIO_stripStdoutBufferContentHeaders()
> >         > result = mapscript.msIO_getStdoutBufferBytes()
> >         > mapscript.msCleanup(1)
> >         > return result, content_type, dispatch_status
> >         >
> >         > Unfortunately we experienced several weird errors
> >         (segfaults, hit assertions
> >         > within GEOS, etc) when dispatching requests, especially in
> >         multi-threaded
> >         > environments (such as a deployment with Apache). We traced
> >         this down and
> >         > found that a call to `msCleanup` right after the OWSDispatch
> >         was causing the
> >         > problems.
> >         >
> >         > Taken from the mapscript documentation:
> >         >
> >         > "msCleanup() attempts to recover all dynamically allocated
> >         resources
> >         > allocated by MapServer code and dependent libraries. It it
> >         used primarily
> >         > for final cleanup in scripts that need to do memory leak
> >         testing to get rid
> >         > of “noise” one-time allocations. It should not normally be
> >         used by
> >         > production code."
> >         >
> >         > When we remove the call to msCleanup, no segfaults and other
> >         errors occur
> >         > within Apache. Unfortunately, while running our test cases
> >         (including
> >         > several hundreds MapServer dispatches) we quickly run out of
> >         memory (e.g:
> >         > memory consumption after 30 test cases was over 2.2 GiB
> >         instead of about 150
> >         > MB).
> >         >
> >         > Any ideas are welcome :)
> >         >
> >         > Thanks,
> >         > Fabian
> >         >
> >         > _______________________________________________
> >         > mapserver-dev mailing list
> >         > mapserver-dev at lists.osgeo.org
> >         > http://lists.osgeo.org/mailman/listinfo/mapserver-dev
> >         _______________________________________________
> >         mapserver-dev mailing list
> >         mapserver-dev at lists.osgeo.org
> >         http://lists.osgeo.org/mailman/listinfo/mapserver-dev
> >
> >
> >
> > _______________________________________________
> > mapserver-dev mailing list
> > mapserver-dev at lists.osgeo.org
> > http://lists.osgeo.org/mailman/listinfo/mapserver-dev
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/mapserver-dev/attachments/20120701/03d8c4d9/attachment.html>