[mapserver-dev] python mapscript and msCleanup issues

[Stephan Meißl]

Thomas, Umberto,

we created a simple (at least as simple as possible) test case. Here is
a basic Python script: http://pastebin.ca/2165345 which we deployed in
Apache using mod_wsgi with this config: http://pastebin.ca/2165346

There is also a basic mapfile using the "road" data in "msautotets/wxs"
http://pastebin.ca/2165329 but I suspect you can use whatever mapfile
you have at hand.

When we issue multiple simultaneous requests e.g. using this script
http://pastebin.ca/2165348 (note the "&" at the end of the curl call)
some of the requests result in errors. 

Either removing the call to msCleanup() or extending the Apache config
to limit the number of threads to 1 (http://pastebin.ca/2165347) seems
to resolve the issues but of course with limitations.

Are we missing anything essential? How should we free request/map
objects without using msCleanup() in Python?

Thanks for any hint,
Stephan


On Thu, 2012-06-28 at 14:15 +0200, Umberto Nicoletti wrote:
> Fabian,
> if you could put together a test case that I can run to reproduce the
> problem I will take a look into it.
> 
> 
> Regards,
> Umberto
> 
> On Thu, Jun 28, 2012 at 1:44 PM, thomas bonfort
> <thomas.bonfort at gmail.com> wrote:
>         Fabian,
>         can you get a backtrace of the segfaults?
>         
>         Aside from that, I would personally refrain from using threads
>         with
>         mapscript and stick to a prefork mpm in apache to accomplish
>         this. It
>         is also my understanding that calling msCleanup() should be
>         limited to
>         when a process is actually exiting, and not at the end of each
>         request, i.e. if I were you I would investigate where memory
>         is
>         leaking when freeing your request/map objects.
>         
>         regards,
>         thomas
>         
>         On Thu, Jun 28, 2012 at 11:28 AM, Fabian Schindler
>         <fabian.schindler at eox.at> wrote:
>         > Hi all,
>         >
>         > Has anyone had issues with Python MapScript and the
>         msCleanup() in
>         > WSGI-Apache applications? Using msCleanup() results in
>         strange
>         > errors/segfaults in multi-threaded setups but not using it
>         quickly eats up
>         > memory.
>         >
>         > We are developing a Django application to serve enhanced WCS
>         responses
>         > (according to the EO-WCS extension). For this purpose, we
>         use MapScript to
>         > receive a basic WCS response, which we are extending with
>         additional
>         > metadata.
>         >
>         > Our mapserver dispatch code is like this:
>         >
>         > mapscript.msIO_installStdoutToBuffer()
>         > dispatch_status = map.OWSDispatch(ows_req)
>         > content_type = mapscript.msIO_stripStdoutBufferContentType()
>         > mapscript.msIO_stripStdoutBufferContentHeaders()
>         > result = mapscript.msIO_getStdoutBufferBytes()
>         > mapscript.msCleanup(1)
>         > return result, content_type, dispatch_status
>         >
>         > Unfortunately we experienced several weird errors
>         (segfaults, hit assertions
>         > within GEOS, etc) when dispatching requests, especially in
>         multi-threaded
>         > environments (such as a deployment with Apache). We traced
>         this down and
>         > found that a call to `msCleanup` right after the OWSDispatch
>         was causing the
>         > problems.
>         >
>         > Taken from the mapscript documentation:
>         >
>         > "msCleanup() attempts to recover all dynamically allocated
>         resources
>         > allocated by MapServer code and dependent libraries. It it
>         used primarily
>         > for final cleanup in scripts that need to do memory leak
>         testing to get rid
>         > of “noise” one-time allocations. It should not normally be
>         used by
>         > production code."
>         >
>         > When we remove the call to msCleanup, no segfaults and other
>         errors occur
>         > within Apache. Unfortunately, while running our test cases
>         (including
>         > several hundreds MapServer dispatches) we quickly run out of
>         memory (e.g:
>         > memory consumption after 30 test cases was over 2.2 GiB
>         instead of about 150
>         > MB).
>         >
>         > Any ideas are welcome :)
>         >
>         > Thanks,
>         > Fabian
>         >
>         > _______________________________________________
>         > mapserver-dev mailing list
>         > mapserver-dev at lists.osgeo.org
>         > http://lists.osgeo.org/mailman/listinfo/mapserver-dev
>         _______________________________________________
>         mapserver-dev mailing list
>         mapserver-dev at lists.osgeo.org
>         http://lists.osgeo.org/mailman/listinfo/mapserver-dev
>         
> 
> 
> _______________________________________________
> mapserver-dev mailing list
> mapserver-dev at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/mapserver-dev