[mapserver-dev] Thread safety [was: python mapscript and msCleanup issues]

[Stephan Meißl]

On Tue, 2012-07-03 at 16:48 +0200, Even Rouault wrote:
> Selon Stephan Meißl <stephan at meissl.name>:
> 
> > Hi all,
> >
> > after some further investigations we think the issue is the non-thread
> > safe usage of VSI IO in WCS 1.1 and 2.0. I created an issue on github
> > [1] and attached a short script to reproduce the problem. As you can see
> > all files are written to "/vsimem/wcsout" which is never cleaned.
> >
> > AFAIK MapServer is meant to be thread save, right? What is the
> > recommended way for the usage of VSI IO? Is there a recommended way to
> > find out process and thread ids and use them in the directory name? Any
> > other ideas?
> 
> You could use the msTmpFile() function that calls msTmpFilename(), which returns
> a unique filename. You are responsible of cleaning files that have been created
> with VSIUnlink(). You can have a look at mapogroutput.c for example.


Even, just seen that I never replied to your mail. Many thanks for your
suggestion following which we successfully resolved the issue (see [1]).

cu
Stephan

[1] https://github.com/mapserver/mapserver/commit/10f619589244fdb8f1192c72c3d9174235addd58

> >
> > Thanks,
> > Stephan
> >
> > [1] https://github.com/mapserver/mapserver/issues/4369
> >
> >
> > On Tue, 2012-07-03 at 07:25 +0200, Umberto Nicoletti wrote:
> > >
> > >
> > > On Monday, July 2, 2012, Stephan Meißl wrote:
> > >         Hi Umberto,
> > >
> > >         many thanks for your help. I tried both suggestions but
> > >         without an
> > >         optimal result.
> > >
> > >         Using msIO_resetHandlers() instead of msCleanup() gives me,
> > >         after using
> > >         a fairly big amount of memory, quite strange results. In fact
> > >         sometimes
> > >         results from previous requests are present in responses. It
> > >         seems, that
> > >         not everything gets cleared. Any suggestion how to investigate
> > >         this?
> > >
> > >
> > > msIO_resetHandlers should probably be invoked after the response has
> > > been commited, could you take a look at this page:
> > >
> > >
> > > http://code.google.com/p/modwsgi/wiki/RegisteringCleanupCode
> > >
> > >
> > > and implement the cleanup hook as described?
> > >
> > >
> > >         Using msIO_getStdoutBufferString() seems to work for text
> > >         responses only
> > >         but not for images.
> > >
> > >         Thanks,
> > >         Stephan
> > >
> > >
> > >         On Sun, 2012-07-01 at 12:06 +0200, Umberto Nicoletti wrote:
> > >         > Before we turn on debugging mode let's try a couple of
> > >         simple changes
> > >         > to your script:
> > >         >
> > >         >
> > >         > http://pastebin.ca/2165984
> > >         >
> > >         >
> > >         > I have replaced the call to msCleanup with a call to
> > >         > msIO_resetHandlers which clears all buffers.
> > >         >
> > >         >
> > >         > http://pastebin.ca/2165986
> > >         >
> > >         >
> > >         > same as before but with the buffer accessed as a string
> > >         rather than a
> > >         > byte array (string is not copied inside mapscript, array
> > >         is).
> > >         >
> > >         >
> > >         > Let us now how this goes.
> > >         >
> > >         >
> > >         > Umberto
> > >         >
> > >         > On Thu, Jun 28, 2012 at 8:02 PM, Stephan Meißl
> > >         <stephan at meissl.name>
> > >         > wrote:
> > >         >         Thomas, Umberto,
> > >         >
> > >         >         we created a simple (at least as simple as possible)
> > >         test
> > >         >         case. Here is
> > >         >         a basic Python script: http://pastebin.ca/2165345
> > >         which we
> > >         >         deployed in
> > >         >         Apache using mod_wsgi with this config:
> > >         >         http://pastebin.ca/2165346
> > >         >
> > >         >         There is also a basic mapfile using the "road" data
> > >         in
> > >         >         "msautotets/wxs"
> > >         >         http://pastebin.ca/2165329 but I suspect you can use
> > >         whatever
> > >         >         mapfile
> > >         >         you have at hand.
> > >         >
> > >         >         When we issue multiple simultaneous requests e.g.
> > >         using this
> > >         >         script
> > >         >         http://pastebin.ca/2165348 (note the "&" at the end
> > >         of the
> > >         >         curl call)
> > >         >         some of the requests result in errors.
> > >         >
> > >         >         Either removing the call to msCleanup() or extending
> > >         the
> > >         >         Apache config
> > >         >         to limit the number of threads to 1
> > >         >         (http://pastebin.ca/2165347) seems
> > >         >         to resolve the issues but of course with
> > >         limitations.
> > >         >
> > >         >         Are we missing anything essential? How should we
> > >         free
> > >         >         request/map
> > >         >         objects without using msCleanup() in Python?
> > >         >
> > >         >         Thanks for any hint,
> > >         >         Stephan
> > >         >
> > >         >
> > >         >         On Thu, 2012-06-28 at 14:15 +0200, Umberto Nicoletti
> > >         wrote:
> > >         >         > Fabian,
> > >         >         > if you could put together a test case that I can
> > >         run to
> > >         >         reproduce the
> > >         >         > problem I will take a look into it.
> > >         >         >
> > >         >         >
> > >         >         > Regards,
> > >         >         > Umberto
> > >         >         >
> > >         >         > On Thu, Jun 28, 2012 at 1:44 PM, thomas bonfort
> > >         >         > <thomas.bonfort at gmail.com> wrote:
> > >         >         >         Fabian,
> > >         >         >         can you get a backtrace of the segfaults?
> > >         >         >
> > >         >         >         Aside from that, I would personally
> > >         refrain from
> > >         >         using threads
> > >         >         >         with
> > >         >         >         mapscript and stick to a prefork mpm in
> > >         apache to
> > >         >         accomplish
> > >         >         >         this. It
> > >         >         >         is also my understanding that calling m
> >
> >
> > _______________________________________________
> > mapserver-dev mailing list
> > mapserver-dev at lists.osgeo.org
> > http://lists.osgeo.org/mailman/listinfo/mapserver-dev
> >
> 
>