[GRASS-dev] lidar tools update in grass7

[Hamish]

Soeren wrote:
> The computation time of overlapping regions should be the
> same in python as in C.
> IMHO it is easier to program a python module which computes
> the tiles, split up the vector points based on the tiles and
> distribute the data across a grid or cloud for computation.
> After the distributed processes are finished, the script can
> gather the data and merge it together.
> 
> If you implement this into the lidar modules, each module
> needs to be changed to support MPI. Using MPI is IMHO much more
> complex, then just running the modules on different datasets
> on different nodes, gathering and patching the data.
...
> Maybe the overlapping algorithm can be simplified in the
> python version?
> In my vision, the LIDAR computation could be run in an
> elastic computation cloud (EC2) as WPS backend using
> recursive tiling algorithm for data distribution.

thinking big!


for my 2c (fwiw I run about 160 computational-hours per day
split up and dynamically sent out to a couple of clusters using
MPI), I would first ask how much processing are we talking
about?  If the typical large-case lidar tools job only took ~3
hours to run on a modern PC, I'd much prefer the simplicity of
OpenMP and run it on the same quad-core desktop or 8-core Xeon
server that the GIS is running on.

If the jobs could take 12-24 hours and have to happen often I
would more seriously consider using MPI. Setting up the mpd
daemon and organizing the mpirun jobs is a lot of overhead,
which for smaller jobs seems slightly counter-productive to me.

also, I think it is quite natural to have the control and
management scripts in python, but get worried about splitting
up low-level processing into a hybrid python/C mix. for one
thing it makes the code harder to follow/understand, for another
you are at risk from different component versions interacting
in different ways. (see wxNviz, vdigit problems on WinGrass..)

(and of course this has to be an optional extra, and I'm all for
exploring all these different methods .. we don't know which
one will win out/survive in a few years time when 16-core
workstations are the norm)


regards,
Hamish