[mapserver-dev] python mapscript and msCleanup issues
Stephan Meißl
stephan at meissl.name
Mon Jul 2 06:08:21 PDT 2012
Hi Umberto,
many thanks for your help. I tried both suggestions but without an
optimal result.
Using msIO_resetHandlers() instead of msCleanup() gives me, after using
a fairly big amount of memory, quite strange results. In fact sometimes
results from previous requests are present in responses. It seems, that
not everything gets cleared. Any suggestion how to investigate this?
Using msIO_getStdoutBufferString() seems to work for text responses only
but not for images.
Thanks,
Stephan
On Sun, 2012-07-01 at 12:06 +0200, Umberto Nicoletti wrote:
> Before we turn on debugging mode let's try a couple of simple changes
> to your script:
>
>
> http://pastebin.ca/2165984
>
>
> I have replaced the call to msCleanup with a call to
> msIO_resetHandlers which clears all buffers.
>
>
> http://pastebin.ca/2165986
>
>
> same as before but with the buffer accessed as a string rather than a
> byte array (string is not copied inside mapscript, array is).
>
>
> Let us now how this goes.
>
>
> Umberto
>
> On Thu, Jun 28, 2012 at 8:02 PM, Stephan Meißl <stephan at meissl.name>
> wrote:
> Thomas, Umberto,
>
> we created a simple (at least as simple as possible) test
> case. Here is
> a basic Python script: http://pastebin.ca/2165345 which we
> deployed in
> Apache using mod_wsgi with this config:
> http://pastebin.ca/2165346
>
> There is also a basic mapfile using the "road" data in
> "msautotets/wxs"
> http://pastebin.ca/2165329 but I suspect you can use whatever
> mapfile
> you have at hand.
>
> When we issue multiple simultaneous requests e.g. using this
> script
> http://pastebin.ca/2165348 (note the "&" at the end of the
> curl call)
> some of the requests result in errors.
>
> Either removing the call to msCleanup() or extending the
> Apache config
> to limit the number of threads to 1
> (http://pastebin.ca/2165347) seems
> to resolve the issues but of course with limitations.
>
> Are we missing anything essential? How should we free
> request/map
> objects without using msCleanup() in Python?
>
> Thanks for any hint,
> Stephan
>
>
> On Thu, 2012-06-28 at 14:15 +0200, Umberto Nicoletti wrote:
> > Fabian,
> > if you could put together a test case that I can run to
> reproduce the
> > problem I will take a look into it.
> >
> >
> > Regards,
> > Umberto
> >
> > On Thu, Jun 28, 2012 at 1:44 PM, thomas bonfort
> > <thomas.bonfort at gmail.com> wrote:
> > Fabian,
> > can you get a backtrace of the segfaults?
> >
> > Aside from that, I would personally refrain from
> using threads
> > with
> > mapscript and stick to a prefork mpm in apache to
> accomplish
> > this. It
> > is also my understanding that calling msCleanup()
> should be
> > limited to
> > when a process is actually exiting, and not at the
> end of each
> > request, i.e. if I were you I would investigate
> where memory
> > is
> > leaking when freeing your request/map objects.
> >
> > regards,
> > thomas
> >
> > On Thu, Jun 28, 2012 at 11:28 AM, Fabian Schindler
> > <fabian.schindler at eox.at> wrote:
> > > Hi all,
> > >
> > > Has anyone had issues with Python MapScript and
> the
> > msCleanup() in
> > > WSGI-Apache applications? Using msCleanup()
> results in
> > strange
> > > errors/segfaults in multi-threaded setups but not
> using it
> > quickly eats up
> > > memory.
> > >
> > > We are developing a Django application to serve
> enhanced WCS
> > responses
> > > (according to the EO-WCS extension). For this
> purpose, we
> > use MapScript to
> > > receive a basic WCS response, which we are
> extending with
> > additional
> > > metadata.
> > >
> > > Our mapserver dispatch code is like this:
> > >
> > > mapscript.msIO_installStdoutToBuffer()
> > > dispatch_status = map.OWSDispatch(ows_req)
> > > content_type =
> mapscript.msIO_stripStdoutBufferContentType()
> > > mapscript.msIO_stripStdoutBufferContentHeaders()
> > > result = mapscript.msIO_getStdoutBufferBytes()
> > > mapscript.msCleanup(1)
> > > return result, content_type, dispatch_status
> > >
> > > Unfortunately we experienced several weird errors
> > (segfaults, hit assertions
> > > within GEOS, etc) when dispatching requests,
> especially in
> > multi-threaded
> > > environments (such as a deployment with Apache).
> We traced
> > this down and
> > > found that a call to `msCleanup` right after the
> OWSDispatch
> > was causing the
> > > problems.
> > >
> > > Taken from the mapscript documentation:
> > >
> > > "msCleanup() attempts to recover all dynamically
> allocated
> > resources
> > > allocated by MapServer code and dependent
> libraries. It it
> > used primarily
> > > for final cleanup in scripts that need to do
> memory leak
> > testing to get rid
> > > of “noise” one-time allocations. It should not
> normally be
> > used by
> > > production code."
> > >
> > > When we remove the call to msCleanup, no segfaults
> and other
> > errors occur
> > > within Apache. Unfortunately, while running our
> test cases
> > (including
> > > several hundreds MapServer dispatches) we quickly
> run out of
> > memory (e.g:
> > > memory consumption after 30 test cases was over
> 2.2 GiB
> > instead of about 150
> > > MB).
> > >
> > > Any ideas are welcome :)
> > >
> > > Thanks,
> > > Fabian
> > >
> > > _______________________________________________
> > > mapserver-dev mailing list
> > > mapserver-dev at lists.osgeo.org
> > >
> http://lists.osgeo.org/mailman/listinfo/mapserver-dev
> > _______________________________________________
> > mapserver-dev mailing list
> > mapserver-dev at lists.osgeo.org
> >
> http://lists.osgeo.org/mailman/listinfo/mapserver-dev
> >
> >
> >
> > _______________________________________________
> > mapserver-dev mailing list
> > mapserver-dev at lists.osgeo.org
> > http://lists.osgeo.org/mailman/listinfo/mapserver-dev
>
>
>
>
>
More information about the mapserver-dev
mailing list