[GRASS-user] RE: [GRASSLIST:1174] Working with very large dat asets

[Patton, Eric]

Just to chime in on Hamish' r.in.xyz - I just got back from a field survey
of the Bay of Fundy where 56GB of swath sonar (Simrad EM1002) was collected.
After exporting each survey day from Caris HIPS as xyz, I've used r.in.xyz +
r.patch to import the entire 56GB into Grass with no problems. I've always
felt that r.in.xyz runs very quickly given the size of each xyz dataset. 

~ Eric. 

-----Original Message-----
From: grassuser-bounces at grass.itc.it
To: Jonathan Greenberg; David Finlayson
Cc: grassuser at grass.itc.it; Helena Mitasova; Andrew at grass.itc.it; Danner
Sent: 8/24/2006 2:50 AM
Subject: Re: [GRASS-user] RE: [GRASSLIST:1174] Working with very large
datasets

David Finlayson wrote:
> I am working with an interferometric sidescan SONAR system that
> produces about 2 Gb of elevation and amplitude data per hour. Our raw
> data density could support resolutions up to 0.1 m, but we currently
> can't handle the data volume at that resolution so we decimate down to
> 1 m via a variety of filters. Still, even at 1 m resolution, our
> datasets run into the hundreds of Mb and most current software just
> doesn't handle the data volumes well. 
>
> Any thoughts on processing and working with these data volumes (LIDAR
> folks)? I have struggled to provide a good product to our researchers
> using both proprietary (Fledermaus, ArcGIS) and non-proprietary (GMT,
> GRASS, my own scripts) post-processing software. Nothing is working
> very well. The proprietary stuff seems easier at first, but becomes
> difficult to automate. The non-proprietary stuff is easy to automate,
> but often can't handle the data volumes without first down sampling
> the data density (GMT does pretty well if you stick to line-by-line
> processing, but that doesn't always work).
>
> Just curious what work flows/software others are using. In particular,
> I'd love to keep the whole process FOSS if possible. I don't trust
> black boxes.