[postgis-users] PostGIS and LIDAR Massive Point Sets

[David Blasby]

collin at socrates.Berkeley.EDU wrote:

> I am participating in a project to improve LIDAR processing.  We have a
> 20 gigabyte dataset of filtered x,y,z points in ascii files.
> 
> I have questions:
> 1. Can PostGIS handle that kind of data at usable speeds? We are using a
> new Sun Blade server running solaris.
> 2. What data types are the point arrays? text? float? Text could cause an
> unusable overhead on a dataset this massive.
> 3. How large a dataset has anyone tried so far? And what have been the
> problems/limitations?
> 
> Any help, commentary, or opinions would be appreciated.


I've been using multi-gigabyte datasets in PostGIS for quite a while and 
havent had any problems.   The speed depends on what you're doing - 
searching 20 million rows for a small-ish bounding box is pretty much 
instant.

The geometry data is stored in its own datastructure.  The details are 
in postgis.h and there's documentation on the website.  There is quite a 
bit of overhead for storing single points.

The structure is holding unused data (offsetX,offsetY,scale, bvol.zmin, 
bvol.zmax, and type should be char), so some of the overhead could be 
minimized.

If this overhead (after you take out the unused data) is an issue, we 
could upgrade the WKB type so it was indexable, and auto-convertable to 
and from the PostGIS types (so you have all the functionality of 
PostGIS).  I discussed this on the list a little while ago. 
Unfortunately, this would be several days of development.  I've been 
wanting someone to either do this or hire us to do it since it would be 
very nice feature.

dave