[GRASS-dev] lidar tools update in grass7

[Soeren Gebbert]

Hello Markus,

2010/4/28 Markus Metz <markus.metz.giswork at googlemail.com>:
> Soeren Gebbert wrote:
>> Hello,
>>
>>>> A while back I did some crude profiling of the v.lidar tools and found
>>>> that it was spending lots and lots of time in the Tcholetsky decomposition
>>>> loop (3-for loops deep).
>>>>
>>>
>>> That's better now because the matrix is a bit smaller.
>>
>> Yes, its a nice symmetric band matrix. Was a bit puzzling for me to
>> understand the creation of the matrix without any documentation!
>
> I added documentation where I understood the code but honestly, the
> inner workins of the lidar_lib are still a mystery to me.

Thanks for adding documentation, that makes maintaining the modules
much easier!
The bilinear and bicubic interpolation is a mystery for me too ... at
module and theoretical level.

>
>> So, i have implemented some conversion functions into the gmath
>> library while moving the tchol* code from lidar lib into the gmath
>> lib
>
> There are still bugs in the code, mostly minor concerning warning and
> error messages, one major in v.surf.bspline for very large raster
> output regions. I'm busy fixing it, also converting it to a new module
> r.resamp.bspline that does particularly well for filling NULL cells, a
> real alternative to (the much slower) r.fillnulls.

I have renamed some variable (italian -> english) in the lidar ilb and
modules, to get a better understanding of what is happening in the
code. I have committed the changes to svn, in case the changes are
wrong, please tell me, i will revert it.

>
>>[...] i would suggest to use MPI parallelization or something else on
>> process base, to concurrently compute the subregions which are created
>> by default. Maybe a python script using overlapping tiling and
>> subprocess  can do the job?
>
> I think that should be integrated into the current lidar tools because
> they make use of the current subregion tiling and C code is still
> faster than Python code (right?). Treatment of overlapping zones of

The computation time of overlapping regions should be the same in
python as in C.
IMHO it is easier to program a python module which computes the tiles,
split up the vector points based on the tiles and distribute the data
across a grid or cloud for computation. After the distributed
processes are finished, the script can gather the data and merge it
together.

If you implement this into the lidar modules, each module needs to be
changed to support MPI. Using MPI is IMHO much more complex, then just
running the modules on different datasets on different nodes,
gathering and patching the data. There are already modules which
support this process (g.region, r.tileset, r.patch ... ). Well, is
there a module which supports the extraction of vector points of the
current region without topology support?

> neighbouring subregions is done sometimes in the lidar_lib, sometimes
> by the modules themselves. It took me quite some time to understand it
> and fix it, therefore I would advise against code porting/duplication.

Maybe the overlapping algorithm can be simplified in the python version?
In my vision, the LIDAR computation could be run in an elastic
computation cloud (EC2) as WPS backend using recursive tiling
algorithm for data distribution.

Best regards
Soeren

>
> My 2c,
>
> Markus M
>