[GRASS-dev] [bug #5252] (grass) Multicore hardware and grass

[Sören Gebbert]

Hi,
there was a discussion on IRC how to parallelize grass to run on 
multicore computers.
We came to the same conclusion as Glynn.
The design of the GRASS core libs
is not thread safe. And to port the core of grass to run in parallel on 
Beowulf clusters or SSI clusters is not practical.

But GRASS can work in parallel already. :)
Just start several modules on several maps and hope the disk IO is
not the limiting factor. And this works on SSI clusters as well.

I think we should think more simple. Since gcc version 4.2, OpenMP is 
available for normal C/C++/Fortran developers without access to 
commercial compilers.
Individual modules can be parallelized for multicore systems with OpenMP 
(http://www.openmp.org).
And i think to work with OpenMP is much easier than posix threads, MPI, 
BSP or SunGrid and stuff.

But the main issues with GRASS and OpenMP are:

1.) You need some parallel programming experience or your programs will
     become slower in the end with OpenMP
2.) To debug multi threaded software is much harder than single thread
     software.
3.) The core libs are not thread safe, so you have to careful choose the
     library functions which are to use in parallel
4.) The data access to the harddisk dont scale in parallel on a single
     disk/conroler system, you will only benfit from OpenMP if you load
     the data to process completely in the memory
5.) ...


So there are only a few modules which will benefit from simple 
parallelizing with OpenMP.
To change this, the core of grass have to be implemented in
a new and different way.

But i think the grass developers have to think about to parallelize
grass. Because the future computer generations will work with multiple
cores (2 - 128?) and the users will expect that programms like grass
will use the multicore power to work. This is not an easy task and needs
a lot of programming knowledge ... well this can be added to the wish
section for grass version 10? :)


Just my two cent
Soeren

Māris Nartišs schrieb:
> Hi,
> 
> I'm not an expert in this area, but as troll 
> (http://en.wikipedia.org/wiki/Internet_troll), I want to add my 0.02.
> 
> Glynn is right - GRASS is not monolite app, that could be tweaked to work in 
> paralell/threads. But individual modules could benefit from paralel 
> processing. 
> 
> And here we come to political decision - how to paralelise it? It's realy 
> important, because GRASS should use one paralelisation mehanism (we don't 
> need one module that works with threads, second - Beowulfs and third works in 
> SSI systems) and there are more than one option. Do we need GRASS to run on 
> clusters? With modifications (like MPI for Beowulf) or by simply launching 
> mupltiple processes (openMosx and other SSI). Or using threads (POSIX or GNU 
> or ?) on slingle machine will be enough?
> 
> IMHO result of such discussion should be a note to GRASS programmers manual  
> like: "When You develop new module to work in paralel, You should use FOO to 
> make it work on/like BAR".
> 
> Maris.
> 
> PS. Some time a go there where efforts to make GRASS run on Beowulf clusters. 
> Is this subproject dead?
> 
> On Thursday 02 November 2006 05:23, Glynn Clements wrote:
>> Request Tracker wrote:
>>> this bug's URL: http://intevation.de/rt/webrt?serial_num=5252
>>>
>>> This time next year dual quad - core desktop systems should be readily
>>> available. What level of effort would it take for GRASS to be able to
>>> split the processing load between the multiple cores?
>> Most of GRASS' processing is in the individual modules, so you would
>> have to re-write individual modules using multi-threaded algorithms.
>>
>> It doesn't help that the core GRASS libraries aren't remotely
>> thread-safe, and can't readily be made so given the nature of the
>> existing API.
> 
> _______________________________________________
> grass-dev mailing list
> grass-dev at grass.itc.it
> http://grass.itc.it/mailman/listinfo/grass-dev
> 
>