[GRASS-user] RE: [GRASSLIST:1174] Working with very large datasets

[Slover, Kevin]

Brandon,
  You can create a shell script to batch import.  Done it many times,
using the r.in.arc command.  Very easy to do, let me know if you need
some help with it.  Same with patching as well, I would basically import
the tiles, then at the end of the script, use the r.patch command.  

Correct me if I am wrong out there GRASS world, might also want to run
r.fillnulls in case the tiles don't line up perfectly??

Kevin Slover
Coastal / GIS Specialist
2872 Woodcock Blvd Suite 230
Atlanta GA 30341
(P) 678-530-0022
(F) 678-530-0044

-----Original Message-----
From: grassuser-bounces at grass.itc.it
[mailto:grassuser-bounces at grass.itc.it] On Behalf Of Brandon M. Gabler
Sent: Thursday, August 24, 2006 2:02 PM
To: grassuser at grass.itc.it
Subject: RE: [GRASS-user] RE: [GRASSLIST:1174] Working with very large
datasets

Speaking of large datasets, I have an extremely large number of ArcInfo
LIDAR
DEM tiles that I want to import into grass and subsequently join
together; is
there a way to batch import files that are all different naming
conventions? I
imagine there is a script somewhere that provides a GUI that allows more
than
one file name input. These files total about 17GB worth of data, and 
leaving as
separate tiles means that they are unmanageable. But importing one at a 
time is
a waste of my time if there is a better way out there (especially if 
there is a
way to import and patch at once?).

Thanks,
Brandon


--
Brandon M. Gabler
Research Associate
Department of Anthropology
1009 E South Campus Drive, Building #30A
University of Arizona
Tucson, AZ 85721
Phone: 520-621-8455
Fax: 520-621-2088


Quoting "Patton, Eric" <epatton at nrcan.gc.ca>:

> Just to chime in on Hamish' r.in.xyz - I just got back from a field
survey
> of the Bay of Fundy where 56GB of swath sonar (Simrad EM1002) was
collected.
> After exporting each survey day from Caris HIPS as xyz, I've used
r.in.xyz +
> r.patch to import the entire 56GB into Grass with no problems. I've
always
> felt that r.in.xyz runs very quickly given the size of each xyz
dataset.
>
> ~ Eric.
>
> -----Original Message-----
> From: grassuser-bounces at grass.itc.it
> To: Jonathan Greenberg; David Finlayson
> Cc: grassuser at grass.itc.it; Helena Mitasova; Andrew at grass.itc.it;
Danner
> Sent: 8/24/2006 2:50 AM
> Subject: Re: [GRASS-user] RE: [GRASSLIST:1174] Working with very large
> datasets
>
> David Finlayson wrote:
>> I am working with an interferometric sidescan SONAR system that
>> produces about 2 Gb of elevation and amplitude data per hour. Our raw
>> data density could support resolutions up to 0.1 m, but we currently
>> can't handle the data volume at that resolution so we decimate down
to
>> 1 m via a variety of filters. Still, even at 1 m resolution, our
>> datasets run into the hundreds of Mb and most current software just
>> doesn't handle the data volumes well.
>>
>> Any thoughts on processing and working with these data volumes (LIDAR
>> folks)? I have struggled to provide a good product to our researchers
>> using both proprietary (Fledermaus, ArcGIS) and non-proprietary (GMT,
>> GRASS, my own scripts) post-processing software. Nothing is working
>> very well. The proprietary stuff seems easier at first, but becomes
>> difficult to automate. The non-proprietary stuff is easy to automate,
>> but often can't handle the data volumes without first down sampling
>> the data density (GMT does pretty well if you stick to line-by-line
>> processing, but that doesn't always work).
>>
>> Just curious what work flows/software others are using. In
particular,
>> I'd love to keep the whole process FOSS if possible. I don't trust
>> black boxes.
>
> _______________________________________________
> grassuser mailing list
> grassuser at grass.itc.it
> http://grass.itc.it/mailman/listinfo/grassuser



_______________________________________________
grassuser mailing list
grassuser at grass.itc.it
http://grass.itc.it/mailman/listinfo/grassuser


Visit Dewberry's website at www.dewberry.com

This email transmission may contain confidential or privileged information. If you receive this email message in error, notify the sender by email and delete the email without reading, copying or disclosing the email contents. The unauthorized use or dissemination of any confidential or privileged information contained in this email is prohibited.  If you are not the intended recipient and intentionally intercept or forward this message to someone else, you may be subject to criminal and/or civil penalties.  See 18 U.S.C. 2511 et seq.