[GRASS-user] v.lidar.edgedetection very slow

[Moritz Lennert]

On 22/06/09 07:00, Michael Perdue wrote:
> 
> On 19-Jun-09, at 8:54 PM, Hamish wrote:
> 
>>
>> Michael wrote:
>>> I actually get the same thing.
>>> For the first while the process runs in full use
>>> of one core, but once the process starts writing results to
>>> the db, the whole process bogs down to a
>>> grinding halt. CPU usage for v.lidar.edgedetection
>>> drops down to ~1% (1% of one core) and sqlite usage rises to
>>> ~16%. Maybe I'm wrong, but my impression was that the
>>> bottle neck was the modifications to the database.
>>
>> hmmm. I've seen no problems on Linux64 + DBF backend. 100% core until
>> completion.
>>
>> Are you running WinXP + SQlite? Can you try with the DBF driver?
>> maybe the common problem is in the SQLite driver...  ??
> 
> I'm running Mac OS X + SQLite with a 64 bit version of grass. I ran a 
> whole bunch of tests today and it looks like there are at least a couple 
> of things going on and SQLite is one of the issues. I ran a 500m x500m 
> tile with three different DB back-ends (SQLite, Postgres and DBF) and 
> these are the results I obtained;
> (SQLite)
> time v.lidar.edgedetection input=Cal_QTile output=Cal_QTile_edges
> real    38m53.458s
> user    1m17.602s
> sys    4m6.353s
> 
> (Postgres)
> time v.lidar.edgedetection input=Cal_QTile output=Cal_QTile_edges
> real    6m49.060s
> user    0m46.622s
> sys    1m20.324s
> 
> (DBF)
> time v.lidar.edgedetection input=Cal_QTile output=Cal_QTile_edges
> real    1m54.065s
> user    0m48.530s
> sys    1m8.686s
> 
> The results with Postgres and SQLite were a real surprise to me.

As Hamish already mentioned, the main bottleneck here is the connection 
to the database which is much slower for actual database systems than 
for simple file-based DBF. Looking at the code, I see lots of calls to 
the Insert(), InsertInterpolation() and UpDate() functions. At each call 
  you are hit with the overhead of the database connection. I don't know 
the code at all, but it might be worth thinking about grouping these 
database calls, possibly by putting all the SQL statements into a temp 
file and executing that only once at the end. In scripts doing this has 
lead to significant speed gains.

Moritz