<html><body>
<p><tt>>> the biggest lidar file used that I know about is Doug's 379GB dataset<br>
>> (14.5 billion points). <br>
>Frightening.</tt><br>
<br>
The above dataset was for two watersheds collected in 2001, the larger of the two watersheds is 9000 square miles . I've recently been working with newer lidar data ( 2007) from a single county with an area of 744 sq. miles ( Craven county, North Carolina, USA) . This county had lidar flown at a submeter posting ( 0.7m? as a guess). The aggregated ASCII x,y,z,intensity file that I created ( for processing using r.in.xyz) from the input las files for that single county was 80 GB .<br>
<br>
I guess my point is that lidar datasets are getting quite massive. If we are going to be working with the lidar data as point data in the GRASS vector framework, go with the most scalable options. Scalability in working with large data sets is a huge benefit in using GRASS over other solutions.<br>
<br>
Doug<br>
<br>
<br>
<br>
Doug Newcomb <br>
USFWS<br>
Raleigh, NC<br>
919-856-4520 ext. 14 doug_newcomb@fws.gov<br>
---------------------------------------------------------------------------------------------------------<br>
The opinions I express are my own and are not representative of the official policy of the U.S.Fish and Wildlife Service or Dept. of Interior. Life is too short for undocumented, proprietary data formats.<br>
<img width="16" height="16" src="cid:1__=0ABBFF77DFDD06C28f9e8a93df938690@fws.gov" border="0" alt="Inactive hide details for Markus GRASS <markus.metz.giswork@googlemail.com>">Markus GRASS <markus.metz.giswork@googlemail.com><br>
<br>
<br>
<table width="100%" border="0" cellspacing="0" cellpadding="0">
<tr valign="top"><td style="background-image:url(cid:2__=0ABBFF77DFDD06C28f9e8a93df938690@fws.gov); background-repeat: no-repeat; " width="40%">
<ul>
<ul>
<ul>
<ul><b><font size="2">Markus GRASS <markus.metz.giswork@googlemail.com></font></b><font size="2"> </font><br>
<font size="2">Sent by: grass-dev-bounces@lists.osgeo.org</font>
<p><font size="2">06/25/2009 03:21 AM</font></ul>
</ul>
</ul>
</ul>
</td><td width="60%">
<table width="100%" border="0" cellspacing="0" cellpadding="0">
<tr valign="top"><td width="1%"><img width="58" height="1" src="cid:3__=0ABBFF77DFDD06C28f9e8a93df938690@fws.gov" border="0" alt=""><br>
<div align="right"><font size="2">To</font></div></td><td width="100%"><img width="1" height="1" src="cid:3__=0ABBFF77DFDD06C28f9e8a93df938690@fws.gov" border="0" alt=""><br>
<font size="2">Hamish <hamish_b@yahoo.com></font></td></tr>
<tr valign="top"><td width="1%"><img width="58" height="1" src="cid:3__=0ABBFF77DFDD06C28f9e8a93df938690@fws.gov" border="0" alt=""><br>
<div align="right"><font size="2">cc</font></div></td><td width="100%"><img width="1" height="1" src="cid:3__=0ABBFF77DFDD06C28f9e8a93df938690@fws.gov" border="0" alt=""><br>
<font size="2">GRASS devel <grass-dev@lists.osgeo.org></font></td></tr>
<tr valign="top"><td width="1%"><img width="58" height="1" src="cid:3__=0ABBFF77DFDD06C28f9e8a93df938690@fws.gov" border="0" alt=""><br>
<div align="right"><font size="2">Subject</font></div></td><td width="100%"><img width="1" height="1" src="cid:3__=0ABBFF77DFDD06C28f9e8a93df938690@fws.gov" border="0" alt=""><br>
<font size="2">Re: [GRASS-dev] vector libs: file based spatial index</font></td></tr>
</table>
<table border="0" cellspacing="0" cellpadding="0">
<tr valign="top"><td width="58"><img width="1" height="1" src="cid:3__=0ABBFF77DFDD06C28f9e8a93df938690@fws.gov" border="0" alt=""></td><td width="336"><img width="1" height="1" src="cid:3__=0ABBFF77DFDD06C28f9e8a93df938690@fws.gov" border="0" alt=""></td></tr>
</table>
</td></tr>
</table>
<br>
<tt>Hamish wrote:<br>
> Moritz wrote:<br>
> <br>
>> The largest file I have used is about 125000 areas with a<br>
>> topo file weighing 42M, so taking your worst estimation,<br>
>> this would mean around 200MB of spatial index, which is<br>
>> still largely acceptable for me.<br>
>> <br>
><br>
> lidar and swath bathymetry data will easily have millions of points,<br>
> and as time goes on this will only expand. I seem to recall that one of<br>
> Radim's big disappointments was that the need to handle this technology/<br>
> data density only really became apparent just when GRASS's new vector<br>
> engine was nearing completion. With some earlier notice it could have<br>
> been designed to scale better. Still, there is much tuning which can<br>
> be done with the present model to reduce the memory overheads, etc.<br>
> <br>
Yes. As an example, for a 2D point dataset, the topo file should be<br>
about 4 times as large as the coor file, same for the spatial index.<br>
This is because each x,y coordinate pair is stored 3 times in the topo<br>
file, plus some other information that is for points not needed, e.g.<br>
area/isle to the left and to the right, start node and end node (start<br>
node = end node for points/centroids). Each x,y coordinate pair is<br>
stored 2 times in the spatial index (rectangle of size zero with N S E W<br>
and N = S, E = W). I see some potential for cleaning up.<br>
> FWIW the sites type (now vector points) in GRASS 4/5 scales well, just<br>
> as much as you can fit in the text file. (not sure if fseeks are 64bit-<br>
> proof there, probably not)<br>
> <br>
I guess that was without topo?<br>
> the biggest lidar file used that I know about is Doug's 379GB dataset<br>
> (14.5 billion points). <br>
Frightening.<br>
> you might look at libLAS (for lidar data -- an OSGeo semi-affiliated<br>
> project: </tt><tt><a href="http://liblas.org/">http://liblas.org/</a></tt><tt> It is my understanding that Howard is<br>
> currently adding spatial index support in the development version.<br>
> You might check out his approach.<br>
> <br>
Will do.<br>
> I have been, and still am ignorant of what advantage a spatial index<br>
> gives you for point data. ... interested to learn why "topology" would<br>
> be useful for points-only data.<br>
> <br>
Strictly speaking, topology and spatial index are two different things,<br>
you could have a spatial index without topo. I can also not see the<br>
usefulness of topology for point data. A spatial index may be useful to<br>
extract a subset (v.select), but in this case you could just as well go<br>
through the points in the coor file, read one at a time and select the<br>
ones that fall into the study area. Should be slower than with a spatial<br>
index but then you're not dragging along the spatial index.<br>
><br>
> In general I'm fairly happy with the no-topology solution for lidar<br>
> data in grass, but a few targeted modules (eg v.info) really need to<br>
> be modified to deal with them.<br>
><br>
><br>
><br>
> Hamish<br>
><br>
><br>
> ps- we still need to hunt through the archives for Radim's posts on these<br>
> issues which explain quite a bit.<br>
> <br>
I remember one comment where he said that the spatial index is not<br>
written out because of time and space concerns. Space should not be an<br>
issue today, and opening an old vector is faster if the spatial index is<br>
available in a file. Of course I would like a solution that needs less<br>
memory and is faster when modifying a spatial index, but I have not the<br>
faintest idea how to do that. Maybe Paul Kelly's tip on memory mapping<br>
can help.<br>
<br>
Markus M<br>
_______________________________________________<br>
grass-dev mailing list<br>
grass-dev@lists.osgeo.org<br>
</tt><tt><a href="http://lists.osgeo.org/mailman/listinfo/grass-dev">http://lists.osgeo.org/mailman/listinfo/grass-dev</a></tt><tt><br>
</tt><br>
</body></html>