[GRASS-dev] Re: parallelizing GRASS modules

[Hamish]

Michael wrote:
> I lost the previous thread but wanted to respond
> to your question about which modules might
> benefit from speedup.
> In our recursive landscape evolution module
> (r.landscape.evol.py), the two GRASS modules that
> take the most time are r.watershed, r.stats, and
> r.walk, especially r.watershed and r.stats since
> we need to run these every model cycle.
> The speedup of r.watershed of a few years back
> made an enormous difference in our model run
> times. But it is still time consuming on
> landscapes with large numbers of cells. If
> parallelization could speed this up, it would be
> great. I'm not sure that r.stats can be
> parallelized or not, but speedup would be helpful.


wrt r.walk: I would think to start with parallelizing r.cost, then porting
the method over to r.walk. Both modules use the segment library, so coding
it to process each segment in its own thread seems like the way to do that.
(and more generally formulate + document a method to parallelize things that use the segment library.
perhaps a simple proof-of-method module to do that should come first)


wrt r.watershed: I guess we'd want a segment mode like the -m flag, but
keeping things in RAM instead of using disk swap? Segment mode does make
use of the segment library..   MarkusM?


wrt r.stats: I suspect it is mostly I/O bound so I don't know how much faster
it can be made, but I ran it through the valgrind profiler anyway just to see.
(see the wiki Bugs page for recipe)

22% of the time is spent in G_quant_get_cell_value()

13% of the time is spent in lib/gis/get_row.c's cell_values_double()

10% of the time is spent in r.stats/stats.c's update_cell_stats() updating a
hash table. may be possible to parallelize that.

if multiple input maps are used perhaps each could be processed in their own
thread, but I don't think you are doing that with LandDyn.

perhaps the way that r.stats is called/used by LandDyn could be refined? ie
is there a lot of unnecessary calculations going on just to get a simple
stat out which could more efficiently be answered in another way? (no idea,
but worth exploring)


Hamish