[GRASS-user] 64 bit or parallel processing

[Jonathan Greenberg]

I'm now working at UCD on a project with > 65 (!) hyperspectral images, and
a LOT of the slow downs stem from I/O intensive processes -- with that said,
there are plenty of "standard" raster processing routines that start
becoming processor intensive (e.g. linear spectral unmixing) that, with a
decent i/o setup (we have a lot of cheap-o 2-drive RAID0s that we back up to
a permanent location, this nearly doubles our I/O speed).  Not having any
idea of how grass's raster access backbone functions, it does seem you could
have a generic tiling routine that could be called from all of the raster
commands that would be highly parallizable (I note that RSI ENVI does this
out of the box now, it detects the # of processors and simply starts forking
a single image across all of the processors, I assume one line/tile per
processor).  Mapcalc and related processes would be a good start, and keep
in mind *spatial* analyses (e.g. texture windows) as you design it.  

My suggestion to the GRASS programmers: if you decide this is useful, I'd
HIGHLY recommend trying to do this in an out-of-the-box approach, e.g.
having an install-time "mp" capability -- if it requires being a) experts in
raster processing, b) experts in GRASS and c) experts in parallel coding,
you'll end up with 3 people doing SMP :)  

--j

-----Original Message-----
From: grassuser-bounces at grass.itc.it [mailto:grassuser-bounces at grass.itc.it]
On Behalf Of Hamish
Sent: Wednesday, February 28, 2007 4:14 PM
To: Glynn Clements
Cc: grassuser at grass.itc.it
Subject: Re: [GRASS-user] 64 bit or parallel processing

Glynn Clements wrote:
> It would be more feasible to modify individual modules; e.g. a
> multi-threaded version of r.mapcalc might be feasible. But unless the
> raster I/O code was made thread-safe, you would still be limited by
> the rate at which map data could be read and written by a single
> thread, so it would only be worthwhile for cases where the bulk of the
> overhead was in the actual calculations.

Right, so working on individual modules which can typically take 10+
hours to run (*.rst, watersheds, viewsheds (incl. r.sun), indices from
RS data, etc) seems like a much better focus of effort for the greatest
gain:work ratio. And as seen in the MPI add-ons this is exactly where
the previous ad-hoc effort has gone.

IIRC Helena said that the RST modules have changed substantially sice
GRASS 5 and the s.surf.rst.mpi work would have to be largely redone.

Any thoughts on gains that could be made by MPI'ing the segmentation
lib? Do do modules using that usually do so for memory needs not
processing speed?


Hamish

_______________________________________________
grassuser mailing list
grassuser at grass.itc.it
http://grass.itc.it/mailman/listinfo/grassuser