RFC 16: MapScript WxS Services

[Frank Warmerdam]

Tamas Szekeres wrote:
> I have only assumptions about Frank's implementation. As far as i know
> he proposed to maintain variables needed to utilize the IO per thread,
> but he would like to create a global repository for theese variables
> requiring the need to synchronize the access between the threads. The
> global repository would be accessed by the thread identifier as a key.

Tamas,

This is exactly correct.  It is also the way that the error contexts
currently work in maperror.c.

> That was the main objection i have mentioned before since it goes
> against the thread independency.

I think my approach to thread-local data (as described above) is not
great, but I do think it is safe and not a significant performance problem.
Once we have real thread-local data by some better mechanism we can pretty
easily replace the existing thread-local handling in maperror.c, and the
similar code I will introduce in mapio.c.

> Well the rule of thumb to achieve this level of operation is simple,
> but to reconstruct the existing code this way is more painful. That's
> why my RFC will be a long lived and continously evolved proposal,
> ensuring this issue to be kept in the foreground. 

I'm not exactly sure how we act on an RFC that is continuously evolving.
This is "new ground" for our RFC approach.

 > Every areas
> identified there requires a detailed analyzation of the code. If we
> have to make a high number of modifications at an area it might be
> better to retain the existing code and keep the access to the global
> resources sychronized or suggesting to avoid that area for the user.
> It must be noted that the current proposal is not the case of the
> "future developments", since it brings in large amount of the existing
> code with minor modifications, and Frank will not want to modify it
> more than it is really needed for the functionality (it is not true
> exactly because he wanted to make the aforementioned threading
> concept).

I am looking at improving the threading support in mapio because it
seems an obvious application of the WxS MapScript services is in
scripting/programming environments that are multi-threaded such as
Java and C#.  Without some reasonable form of thread control the
utility of RFC 16 would be partially neutered.

> In this case we have to consider the following issues:
> 
> 1. The current WxS implementation has a large amount of the existing
> code would be difficult to recreate from scratch.

Amen!

> 2. The current WxS implementation comes from the cgi world having the
> (global) standard output as the primary output interface. Therefore
> the current msIO invocations do not have an additional parameter for
> passing back the return value (io context in our previous terminology)

Right

> 3. Using this additional parameter would require a large amount of
> change, and we should propagate this parameter through the entire call
> stack until the OWSDispatch call. I did not consider if this context
> should be go outside of the mapscript call if the result is created
> within the same call and an appropriate output structure can be filled
> (gdBuffer was mentioned for this purpose). With this issue i was a bit
> optimistic because the io context could be implemented easily at the
> mapio.c side, but the large number of invocations makes this work much
> more complicated.

I have been thinking of io somewhat more globally than just OWSDispatch
and functions called from it.  Perhaps it would be possible to introduce
io contexts in *just* the OWS services without having it affect the more
traditional msDrawMap() and related code ... but I'm not at all certain of
that.  My intent with RFC 16, though not explicitly stated, is that the
io redirection would work with anything to stdout including stuff like
MapServer's "traditional" template processing output.

It is certainly possible that a targetted introduction of io contexts
would be possible, and desirable, in just the OWSDispatch chain though
even at that quite a bit of code would have to be touched.

> Frank,
> 
> Have you consider to implement a real thread local io context using
> the TLS api?

Well, as I mentioned earlier, I'm not sure there is a single "TLS" API
avaiable across platforms currently.  So we would either need to
construct this within mapserver using appropriate platform specific
mechanisms (essentially win32 or pthread) or we would need to use some
support library for this service (potentially even GDAL!).

Once that is done it will be trivial to modify mapio.c and maperror.c
to use the TLS mechanism instead of the current approach.

 >  If we could create the variables collecting the output
> stream as thread local at the beginning of the OWSDispatch call and we
> would not want to retain this variable between the OWSDispatch calls
> we could have an equivalent implementation as passing the local
> contexts between the function calls. As far as i know the TLS api does
> not involve unnecessary synchronizations between the threads.

To the best of my knowledge, the various TLS API's internally use a mutex
to manage access to a list of thread local datastructures much as I do.
The benefit of a TLS API is that it is likely implemented better by the
operating system folks, and it is abstracted from us.  But really, many
OS services already use mutexes within them to work properly in
multi-threaded environments, including every malloc/free, and printf call
when using thread-safe C libraries.

Best regards,
-- 
---------------------------------------+--------------------------------------
I set the clouds in motion - turn up   | Frank Warmerdam, warmerdam at pobox.com
light and sound - activate the windows | http://pobox.com/~warmerdam
and watch the world go round - Rush    | President OSGF, http://osgeo.org