[mapserver-dev] Thread safety [was: python mapscript and msCleanup issues]

[Stephan Meißl]

Hi all,

after some further investigations we think the issue is the non-thread
safe usage of VSI IO in WCS 1.1 and 2.0. I created an issue on github
[1] and attached a short script to reproduce the problem. As you can see
all files are written to "/vsimem/wcsout" which is never cleaned.

AFAIK MapServer is meant to be thread save, right? What is the
recommended way for the usage of VSI IO? Is there a recommended way to
find out process and thread ids and use them in the directory name? Any
other ideas?

Thanks,
Stephan

[1] https://github.com/mapserver/mapserver/issues/4369


On Tue, 2012-07-03 at 07:25 +0200, Umberto Nicoletti wrote:
> 
> 
> On Monday, July 2, 2012, Stephan Meißl wrote:
>         Hi Umberto,
>         
>         many thanks for your help. I tried both suggestions but
>         without an
>         optimal result.
>         
>         Using msIO_resetHandlers() instead of msCleanup() gives me,
>         after using
>         a fairly big amount of memory, quite strange results. In fact
>         sometimes
>         results from previous requests are present in responses. It
>         seems, that
>         not everything gets cleared. Any suggestion how to investigate
>         this?
> 
> 
> msIO_resetHandlers should probably be invoked after the response has
> been commited, could you take a look at this page:
> 
> 
> http://code.google.com/p/modwsgi/wiki/RegisteringCleanupCode
> 
> 
> and implement the cleanup hook as described?
>  
>         
>         Using msIO_getStdoutBufferString() seems to work for text
>         responses only
>         but not for images.
>         
>         Thanks,
>         Stephan
>         
>         
>         On Sun, 2012-07-01 at 12:06 +0200, Umberto Nicoletti wrote:
>         > Before we turn on debugging mode let's try a couple of
>         simple changes
>         > to your script:
>         >
>         >
>         > http://pastebin.ca/2165984
>         >
>         >
>         > I have replaced the call to msCleanup with a call to
>         > msIO_resetHandlers which clears all buffers.
>         >
>         >
>         > http://pastebin.ca/2165986
>         >
>         >
>         > same as before but with the buffer accessed as a string
>         rather than a
>         > byte array (string is not copied inside mapscript, array
>         is).
>         >
>         >
>         > Let us now how this goes.
>         >
>         >
>         > Umberto
>         >
>         > On Thu, Jun 28, 2012 at 8:02 PM, Stephan Meißl
>         <stephan at meissl.name>
>         > wrote:
>         >         Thomas, Umberto,
>         >
>         >         we created a simple (at least as simple as possible)
>         test
>         >         case. Here is
>         >         a basic Python script: http://pastebin.ca/2165345
>         which we
>         >         deployed in
>         >         Apache using mod_wsgi with this config:
>         >         http://pastebin.ca/2165346
>         >
>         >         There is also a basic mapfile using the "road" data
>         in
>         >         "msautotets/wxs"
>         >         http://pastebin.ca/2165329 but I suspect you can use
>         whatever
>         >         mapfile
>         >         you have at hand.
>         >
>         >         When we issue multiple simultaneous requests e.g.
>         using this
>         >         script
>         >         http://pastebin.ca/2165348 (note the "&" at the end
>         of the
>         >         curl call)
>         >         some of the requests result in errors.
>         >
>         >         Either removing the call to msCleanup() or extending
>         the
>         >         Apache config
>         >         to limit the number of threads to 1
>         >         (http://pastebin.ca/2165347) seems
>         >         to resolve the issues but of course with
>         limitations.
>         >
>         >         Are we missing anything essential? How should we
>         free
>         >         request/map
>         >         objects without using msCleanup() in Python?
>         >
>         >         Thanks for any hint,
>         >         Stephan
>         >
>         >
>         >         On Thu, 2012-06-28 at 14:15 +0200, Umberto Nicoletti
>         wrote:
>         >         > Fabian,
>         >         > if you could put together a test case that I can
>         run to
>         >         reproduce the
>         >         > problem I will take a look into it.
>         >         >
>         >         >
>         >         > Regards,
>         >         > Umberto
>         >         >
>         >         > On Thu, Jun 28, 2012 at 1:44 PM, thomas bonfort
>         >         > <thomas.bonfort at gmail.com> wrote:
>         >         >         Fabian,
>         >         >         can you get a backtrace of the segfaults?
>         >         >
>         >         >         Aside from that, I would personally
>         refrain from
>         >         using threads
>         >         >         with
>         >         >         mapscript and stick to a prefork mpm in
>         apache to
>         >         accomplish
>         >         >         this. It
>         >         >         is also my understanding that calling m