[postgis-users] Massive Lidar Dataset Datatype Suggestions?

[Gerry Creager N5JXS]

Collin,

I've nothing to add that will help you, but I'd be interested in hearing 
the results.  I expect to be working with similar datasets *very* soon 
in a coastal and ocean observing project, associated with DEM generation 
for inundation projections.

I suspect there are some issues with memory in your install.  I'm 
implementing a 3GHz Xeon 4-way SMP with 16 GB RAM for our PostGIS 
implementation... a pair of 733 MHz processors with 1 GB RAM seems a 
little light.  Also, have you considered a RAID 1 system for mirroring, 
to improve data integrity?

Please keep me in the loop if at all possible.  We *are* interested.

Regards,
Gerry

collin wrote:
> Greetings all,
> 
> I asked some questions about massive point datasets in April, before the 
> implementation of LWGEOM. Then I fell off the face of the earth for a 
> while.  Know I'm back working on this issue again and would love some 
> input.
> 
> I have a LIDAR dataset of 473 million points with 2 point geometries 
> (first & last returns), timestamp and 2 laser intensity returns (integer).
> 
> I am trying to figure the best setup for storing, extracting and 
> processing this dataset.  btw, it is a smallish dataset. We will be 
> processing 2 billion+ point projects in the near future.
> 
> Currently I compiled & installed postgresql 8.0 beta 4, with postgis 0.9 
> release, geos 2.0.1, and proj 4.  This is on Fedora Core 2, smp 733mhz, 
> 1GB ram, 160gb hdd Intellistation.
> 
> I am using HWGEOM, with WKT right now and managed to create a table with 
>  oid & the_geom for one point per return.
> The upload took 48 hours and is roughly 85GB.
> The GiST indexing took ~80 hours and is ~35GB.
> 
> This is obviously non-optimal considering we now have WKB and LWGEOM to 
> play with.  I couldn't get LWGEOM to install properly from the cvs 
> extract, which is why I reverted to the 0.9 version.
> 
> So, any suggestions on how to get the full 9-column dataset uploaded 
> with a more efficient data structure?  (note: current machine is just a 
> test machine. Production will have a LOT more drivespace).
> 
> Also, I intend to perform a fair amount of point processing inside the 
> database using either plpgsql or java api.  Is this a bad idea?
> 
> Thanks for any input.
> 
> ________________
> Collin Bode
> GIS Informatics Researcher
> Power Lab, Integrative Biology
> University of California, Berkeley
> _______________________________________________
> postgis-users mailing list
> postgis-users at postgis.refractions.net
> http://postgis.refractions.net/mailman/listinfo/postgis-users

-- 
Gerry Creager -- gerry.creager at tamu.edu
Texas Mesonet -- AATLT, Texas A&M University	
Cell: 979.229.5301 Office: 979.458.4020 FAX: 979.847.8578
Page: 979.228.0173
Office: 903A Eller Bldg, TAMU, College Station, TX 77843