[GRASS-dev] [SoC] Parallelization of Raster and Vector modules

[Doug_Newcomb at fws.gov]

>>Jordan,
>>I've dealt with ERDAS Imagine files larger than 10 GB on a regular 
basis.  I have occasionally tried to >>reproject and merge all of the 1 m 
NAIP imagery tiles  for North Carolina into 1 BigTIFF > 500GB with gdal. 
>> Any parallelizaion work for open source geospatial tools would be 
welcome :-).
>So you "tried" to do that? Does that mean you failed or is the system 
still processing it? :-P I kid, but that is a very large file... it's 
bigger than my current desktop's hard drive. 
Jordan,
It would be a shame to leave those multicore  64 bit processors with 
multi-terabyte hard drives with nothing to challange them.  Just think of 
me as part of the hardware entertainment committee. :-) 
The process ran overnight and finished, I just have not gotten to phase 2 
of that particular project, chopping the resulting image into overlapping 
DOQQ - size images.  I was going to go back and change the projection 
method slightly, hopefully to make it more accurate.  If you're looking 
for big images to play with, go to http://datagateway.nrcs.usda.gov/  and 
select a naip county mosaic of imagery.  The mosaic comes as a MrSid 
format file, but once you uncompress it ( I suggest using gdal's nearblack 
utility ) to ERDAS Imagine format, the resulting image will probably be in 
the multi-gigbyte range per county.

Doug
 

Doug Newcomb 
USFWS
Raleigh, NC
919-856-4520 ext. 14 doug_newcomb at fws.gov
---------------------------------------------------------------------------------------------------------
The opinions I express are my own and are not representative of the 
official policy of the U.S.Fish and Wildlife Service or Dept. of the 
Interior.   Life is too short for undocumented, proprietary data formats.



Jordan Neumeyer <jordan.neumeyer at mines.sdsmt.edu> 
Sent by: grass-dev-bounces at lists.osgeo.org
04/08/2010 01:09 AM

To
Doug_Newcomb at fws.gov
cc
GRASS developers list <grass-dev at lists.osgeo.org>
Subject
Re: [GRASS-dev] [SoC] Parallelization of Raster and Vector modules






Thanks guys for putting that into perspective.

On Mon, Apr 5, 2010 at 6:38 PM, <Doug_Newcomb at fws.gov> wrote:

Doug

Doug Newcomb
USFWS
Raleigh, NC
919-856-4520 ext. 14 doug_newcomb at fws.gov
---------------------------------------------------------------------------------------------------------
The opinions I express are my own and are not representative of the 
official policy of the U.S.Fish and Wildlife Service or Dept. of the 
Interior.   Life is too short for undocumented, proprietary data formats.

-----grass-dev-bounces at lists.osgeo.org wrote: -----

To: Jordan Neumeyer <jordan.neumeyer at mines.sdsmt.edu>
From: Markus Neteler <neteler at osgeo.org>
Sent by: grass-dev-bounces at lists.osgeo.org
Date: 04/05/2010 04:06AM
cc: GRASS developers list <grass-dev at lists.osgeo.org>
Subject: Re: [GRASS-dev] [SoC] Parallelization of Raster and Vector 
modules

On Sun, Apr 4, 2010 at 3:22 AM, Jordan Neumeyer
<jordan.neumeyer at mines.sdsmt.edu> wrote:
> I didn't realize how big the data set could be. What's
> biggest map you've seen?

Our provincial DEM is a 3.5GB Geotiff which is of 48800x58000 size.
Another file which I recently had to import was a 4GB Geotiff with
21550 bands. Finally, in remote sensing, you can quickly generate
quickly files in the multi-GB range.

Markus
_______________________________________________
grass-dev mailing list
grass-dev at lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

~Jordan 
_______________________________________________
grass-dev mailing list
grass-dev at lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.osgeo.org/pipermail/grass-dev/attachments/20100412/f4d60223/attachment.html