[GRASS-dev] Using GRASS in long running and multithreaded applications Was: Re: The tomcat shut down ...

[Soeren Gebbert]

Was: Re: The tomcat shut down when encounter some error grass commond

2009/9/24 Glynn Clements <glynn at gclements.plus.com>:
>
> Soeren Gebbert wrote:
>
>> > It isn't just that it's a lot of work to change it, but that it
>> > potentially causes problems for code which uses the libraries (e.g.
>> > simply making the variables non-"static" pollutes the global
>> > namespace).
>>
>> Sorry, i did not quite understand what "pollute the global namespace"
>> exactly means,
>> well i am not a native speaker and not a C expert, so can you please
>> provide an example? :)
>
> If two libraries both export a symbol named e.g. "name", you'll get a
> conflict if you use both libraries from the same program. Depending
> upon the platform, you may get an error, or you may end up with the
> variables being merged into a single variable.
>
> This isn't a problem if the variable isn't exported (i.e. declared
> with the "static" qualifier). But you have can't use "static" if the
> variable needs to be accessible in more than one file.

Thanks for the explanation, now i understand.

>
>> > That could be gotten around by moving all of the static data for each
>> > library into a single "state" variable. That in itself isn't an issue,
>> > but you have to remain vigilant for developers adding new static
>> > variables ad-hoc.
>>
>> An other approach might be the implementation of reset functions in
>> every source file which uses
>> static variables. The reset function can be called  for a fine grain
>> reset approach or all together
>> in a single reset function which collects all distributed reset functions.
>>
>> i.e:
>> New reset functions like:
>>
>> G_reset_geodesic_equation in lib/gis/geodesic.c
>> G_reset_window in lib/gis/get_window.c
>> G_reset_cell_area_calculations in lib/gis/area.c
>>
>> And so on ...
>
> That will get around the namespace issue.
>
> However, it doesn't really help with multithreading. For that, you

Indeed.

> would want to store all of the static data in a library in a state
> structure, and give each thread a pointer to a separate state
> structure (e.g. pthread_setspecific).

IMHO this approach will result in a re-design and re-write of most of
the grass libraries
and an update of more than 300 modules ... .
Looks like a 2 1/2 years full time job for one developer ... any
sponsors available? :)

I think implementing the reset functions is the first step to use
grass in long running applications.
Within those application it must be assured that all the grass library
functions are called from
within the same thread, using a producer-consumer pattern.
Reading and writing of raster and vector maps is performed in serial
by the producer thread
using a queue for all requests from consumer threads. The producer
thread will call the reset functions after each
processed request. A consumer thread will always request full raster
and vector maps. :/
So the processing of the data read into the memory will be independent
from the producer thread.
Read and write calls are queued and the consumer threads need to wait
for the producer thread.

Well, that sounds quite complex.

Glynn, if i remember correctly you described a similar approach for a
multi-threaded raster library
some time ago. Are there any plans to implement such an design in the
grass7 raster lib?
I have no experience with such an approach, but i would like to
realize it in a Java application ... someday. :)

>
>> It would also be great to open raster maps without setting the global
>> window variable
>> but providing a pointer of a region to the opening function?
>> Glynn do you think this is possible without an overhead of work on
>> other functions?
>
> The main problem here is that the mapping between the region grid and
> the raster grid is recalculated whenever you change the region, so
> that all maps are read and written at a single resolution. That isn't
> a particularly common operation, but it is used (e.g. r.resamp.*) and
> needs to be supported in some way.

I see.

>
> This is one area which could do with an overhaul, but it's a
> significant architectural change rather than modifying a few
> functions.

Thanks for your estimation. I will have a look at the raster io code
to evaluate the effort and which function must be changed or
new implemented without breaking the compatibility.

Best regards
Soeren

>
> --
> Glynn Clements <glynn at gclements.plus.com>
>