[GRASS-user] v.lidar.edgedetection very slow

[Hamish]

Michael P wrote:
> I'm running Mac OS X + SQLite with a 64 bit version of
> grass. I ran a whole bunch of tests today and it looks like
> there are at least a couple of things going on and SQLite is
> one of the issues. I ran a 500m x500m tile with three
> different DB back-ends (SQLite, Postgres and DBF) and these
> are the results I obtained;
> (SQLite)
> time v.lidar.edgedetection input=Cal_QTile
> output=Cal_QTile_edges
> real    38m53.458s
> user    1m17.602s
> sys    4m6.353s
> 
> (Postgres)
> time v.lidar.edgedetection input=Cal_QTile
> output=Cal_QTile_edges
> real    6m49.060s
> user    0m46.622s
> sys    1m20.324s
> 
> (DBF)
> time v.lidar.edgedetection input=Cal_QTile
> output=Cal_QTile_edges
> real    1m54.065s
> user    0m48.530s
> sys    1m8.686s
> 
> The results with Postgres and SQLite were a real surprise
> to me.

I am surprised that all DBs spend so much time in "sys" kernel
land. (a good guess there is the time is spent on I/O with disk
controller)  I knew that linux's IO was cleaner than OSX's,
but I didn't know the margin was that much ...

I just ran a test in 64bit linux got the following:

DBF:
real    32m14.192s
user    32m6.880s
sys     0m5.884s

Sqlite:
real    39m45.760s
user    34m25.357s
sys     0m23.225s

I would note that the number of passes it takes to complete
varies between ~ 9 and 20 for my test data depending on
settings, and that affects the time to complete hugely.
(1 pass per subregion, but it skips some number of no-points
subregions)

By watching "top" and gkrellm monitor* I can see that the drop
down to single digit processor use for sqlite happens in the
"Point classification" step, and gkrellm shows that during this
time the disk write is about 22mb/sec, so lets say it is limited
by writing to the disk. After it finishes the point class. step
it jumps back up to 100% for the next pass's interpolations.
(dominated by Tcholetsky decomposition function) Also subsequent
passes often seem to have many fewer points to classify, so
are faster to complete that stage.

[*] something similar to gkrellm can be found in the system
monitor app in the OSX Apps/Utility folder


So DBF is faster to write down a zillion new records to disk.
not all that surprising seeing that it has so little overhead
to worry about.


Hamish:
> > if you build from the latest grass 6.5 or 7 svn you
> > can use the --verbose flag to follow the action. setting
> > 'g.gisenv set="DEBUG=1" gives you even more detail (even
> > with existing versions).
> 
> Does this print out the debugging messages that are in the
> code

yes, you just need to switch them on and then you can see what's
happening. already present in all 6.4.0 RC versions.
set it back to "0" to turn debug messages off again.


> or do I need to do something else to see those?

in the devel versions (6.5 and 7) I've moved the milepost
debug messages into verbose messages, so you just need to
add --verbose to the command line to see them. percent complete
too. It's a lot cleaner than the debug messages, but you need to
recompile from source to get this.



Hamish