<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:st1="urn:schemas-microsoft-com:office:smarttags" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv=Content-Type content="text/html; charset=us-ascii">
<meta name=Generator content="Microsoft Word 11 (filtered medium)">
<!--[if !mso]>
<style>
v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style>
<![endif]--><o:SmartTagType
namespaceuri="urn:schemas-microsoft-com:office:smarttags" name="PostalCode"/>
<o:SmartTagType namespaceuri="urn:schemas-microsoft-com:office:smarttags"
name="State"/>
<o:SmartTagType namespaceuri="urn:schemas-microsoft-com:office:smarttags"
name="PlaceType"/>
<o:SmartTagType namespaceuri="urn:schemas-microsoft-com:office:smarttags"
name="PlaceName"/>
<o:SmartTagType namespaceuri="urn:schemas-microsoft-com:office:smarttags"
name="City"/>
<o:SmartTagType namespaceuri="urn:schemas-microsoft-com:office:smarttags"
name="place"/>
<!--[if !mso]>
<style>
st1\:*{behavior:url(#default#ieooui) }
</style>
<![endif]-->
<style>
<!--
/* Font Definitions */
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman";}
a:link, span.MsoHyperlink
{color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{color:purple;
text-decoration:underline;}
p
{mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:12.0pt;
font-family:"Times New Roman";}
span.EmailStyle17
{mso-style-type:personal-reply;
font-family:Arial;
color:navy;}
@page Section1
{size:8.5in 11.0in;
margin:1.0in 1.25in 1.0in 1.25in;}
div.Section1
{page:Section1;}
-->
</style>
<!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang=EN-US link=blue vlink=purple>
<div class=Section1>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'>David:<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'> We’ve
worked with a ~40gb pansharpened 1m image of the Lake Tahoe Basin using RSI
ENVI – ENVI will support essentially unlimited file sizes on Windows and
many unix boxes, and the next version (4.3) will have LFS on MacOS X as well. I
honestly don’t know how GRASS handles big datasets (I’m sure
someone will respond), but a “good” algorithm basically just
performs the processing using subsets of the data – ENVI processes images
on a per-line basis, so you never really have much of a memory hit, although it
clearly takes a long time to process a 40gb file. ESRI products are
completely useless for large files, in fact I’m pretty sure they simply
can’t deal with any file > 2gb, and their routines are VERY
inefficient. The other issues are whether an OS can actually open a large
file (e.g. MacOS X pre-tiger could not), and how easy it is to use an MP system
(e.g. ENVI will just use a MP system out of the box, but I don’t think
GRASS can).<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'> If
you have some $$$ for hardware, I/O with image processing is pretty important
as well – for a small system, we found a RAID0 “scratch”
drive was a good addition – the extremely high I/O really helps processing
low CPU algorithms (e.g. basic raster calculations). It’s also very
unstable (one drive failure will cause data failure across all drives in the
RAID), so you do have to be careful to backup your system.<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'><o:p> </o:p></span></font></p>
<p class=MsoNormal style='text-indent:.5in'><font size=2 color=navy face=Arial><span
style='font-size:10.0pt;font-family:Arial;color:navy'>Anyway, my two cents!
Along these lines, how DOES GRASS do raster processing? I feel like it
does use tiled processing like I described for mapcalc and most of the other
algorithms – I never see any major forks in memory usage. Has
anyone gotten GRASS working with an MP setup for things like mapcalc?<o:p></o:p></span></font></p>
<p class=MsoNormal style='text-indent:.5in'><font size=2 color=navy face=Arial><span
style='font-size:10.0pt;font-family:Arial;color:navy'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'>--j<o:p></o:p></span></font></p>
<div>
<p><font size=2 color=navy face="Times New Roman"><span style='font-size:10.0pt;
color:navy'>--<br>
<br>
Jonathan A. Greenberg, PhD<br>
NASA Postdoctoral Researcher<br>
<st1:place w:st="on"><st1:PlaceName w:st="on">NASA</st1:PlaceName> <st1:PlaceName
w:st="on">Ames</st1:PlaceName> <st1:PlaceName w:st="on">Research</st1:PlaceName>
<st1:PlaceType w:st="on">Center</st1:PlaceType></st1:place><br>
MS 242-4<br>
<st1:place w:st="on"><st1:City w:st="on">Moffett Field</st1:City>, <st1:State
w:st="on">CA</st1:State> <st1:PostalCode w:st="on">94035-1000</st1:PostalCode></st1:place><br>
Phone: 415-794-5043<br>
AIM: jgrn3007<br>
MSN: jgrn3007@hotmail.com</span></font><font color=navy><span style='color:
navy'> </span></font><o:p></o:p></p>
</div>
<div>
<div class=MsoNormal align=center style='text-align:center'><font size=3
face="Times New Roman"><span style='font-size:12.0pt'>
<hr size=2 width="100%" align=center tabindex=-1>
</span></font></div>
<p class=MsoNormal><b><font size=2 face=Tahoma><span style='font-size:10.0pt;
font-family:Tahoma;font-weight:bold'>From:</span></font></b><font size=2
face=Tahoma><span style='font-size:10.0pt;font-family:Tahoma'> owner-GRASSLIST@baylor.edu
[mailto:owner-GRASSLIST@baylor.edu] <b><span style='font-weight:bold'>On Behalf
Of </span></b>David Finlayson<br>
<b><span style='font-weight:bold'>Sent:</span></b> Monday, August 07, 2006
10:36 PM<br>
<b><span style='font-weight:bold'>To:</span></b> GRASS Users List<br>
<b><span style='font-weight:bold'>Subject:</span></b> [GRASSLIST:1174] Working
with very large data sets</span></font><o:p></o:p></p>
</div>
<p class=MsoNormal><font size=3 face="Times New Roman"><span style='font-size:
12.0pt'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=3 face="Times New Roman"><span style='font-size:
12.0pt'>I am working with an interferometric sidescan SONAR system that
produces about 2 Gb of elevation and amplitude data per hour. Our raw data
density could support resolutions up to 0.1 m, but we currently can't handle
the data volume at that resolution so we decimate down to 1 m via a variety of
filters. Still, even at 1 m resolution, our datasets run into the hundreds of
Mb and most current software just doesn't handle the data volumes well. <br>
<br>
Any thoughts on processing and working with these data volumes (LIDAR folks)? I
have struggled to provide a good product to our researchers using both
proprietary (Fledermaus, ArcGIS) and non-proprietary (GMT, GRASS, my own
scripts) post-processing software. Nothing is working very well. The proprietary
stuff seems easier at first, but becomes difficult to automate. The
non-proprietary stuff is easy to automate, but often can't handle the data
volumes without first down sampling the data density (GMT does pretty well if
you stick to line-by-line processing, but that doesn't always work). <br>
<br>
Just curious what work flows/software others are using. In particular, I'd love
to keep the whole process FOSS if possible. I don't trust black boxes.<br>
<br>
Cheers,<br>
<br>
-- <br>
David Finlayson <o:p></o:p></span></font></p>
</div>
</body>
</html>