[GRASS-dev] Using GRASS in long running and multithreaded applications Was: Re: The tomcat shut down ...

[Soeren Gebbert]

Hello Hamish,

2009/9/26 Hamish <hamish_b at yahoo.com>:
> Glynn wrote:
>> The main issue for concurrent reading is that the raster
>> library caches the current row, so that upscaling doesn't
>> read and decode each row multiple times. That's problematic
>> if you want multiple threads reading the same map.
>
> to me multi-threaded multi-map read does not seem so exciting.
> typically both maps are in the same GISDBASE, so on the part of
> the same disk drive, which can only physically read one thing
> at a time, and asking the drive to jump around non-sequentially
> to read two maps at once would just slow the overall read down.
> I've got nothing against making the libs thread-safe, but
> personally I'm much more intersted in preping the processor-bound
> modules for multi-threading as I see more gains there vs a
> slight speedup in I/O-bound libs.

I think that some algorithms are only parallelizable if the I/O runs
in parallel too.
Especially in case that the algorithms are based on row processing.
Otherwise you need to read huge parts of the raster maps in RAM to benefit from
multithreading.
The gpde and gmath libraries are already partly multi threaded with OpenMP.
Especially the blas level 2 and 3 functions and the krylov-subspace
linear equation solver (cg, pcg and bicgstab).

But you will only have a benefit from multi threading in case  the
data you like to process is large enough and the compiler provides
excellent OpenMP support (intel!).
If the data is not large enough the time the threads needs to spawn may take
longer then processing the data (in case you are not using a thread pool).

Most of the time you need to rewrite the algorithm to enable good speedup
on multiple threads, thats the experience i have.

My hope is that grass will benefit from the parallel VTK code using
the vtkGRASSBridge.
The VTK library ships many multithreaded and MPI parallelized image
and vector processing algorithms. AFAIK they support the processing of
tiles on a cluster.

Best regards
Soeren

>
> but then I'm not trying to keep the libs running long term :)
> and don't know how the vector cleaning tasks fit into this.
>
>
> Hamish
>
>
>
>
>