<html><body>
<p><tt><br>
>Hi Doug,<br>
<br>
>I finally tried your parameters and they did work fine for me also. I<br>
>had something like hundred geotiffs, 400 MB each, and I was pushing<br>
>them to bigtiff mosaic. I tried first with your *.tif selection and then<br>
>again by using a virtual raster file as source, created from Mapserver<br>
>tileindex shapefile with gdalbuildvrt. My Windows computer was handling<br>
>about 20 GB/hour with cubic resampling (-rc) this time. Parameters <br>
?-wo "SKIP_NOSOURCE" --config "GDAL_CACHEMAX=500" -wm=5000<br>
>seem to have a big influence on efficiency. I wonder if there are some<br>
>rules of thumb for selecting values of GDAL_CACHEMAX and -wm. You said<br>
>cachemax is good to be close to the maximum input file size, how about<br>
>-wm?<br>
<br>
>-Jukka Rahkonen-</tt><br>
<br>
<tt>Jukka,</tt><br>
<tt> I remember seeing someone mention on the mailing list ( Can't recall who at the moment) that setting GDAL_CACHEMAX close to the maximum size of the input files gave the best performance. I did try bumping GDAL_CACHEMAX up to 2000 to see what would happen ( while dropping -wm down to 3000, only 6GB of RAM on that computer) , but none of the input files I was processing were larger than 500MB and I saw no increase in performance. For the -wm parameter, I just gave it the rest of the RAM available on the computer, and I did not benchmark while varying that number.</tt><br>
<br>
<tt>Doug</tt><br>
<tt> </tt><br>
<br>
Doug Newcomb <br>
USFWS<br>
Raleigh, NC<br>
919-856-4520 ext. 14 doug_newcomb@fws.gov<br>
---------------------------------------------------------------------------------------------------------<br>
The opinions I express are my own and are not representative of the official policy of the U.S.Fish and Wildlife Service or Dept. of the Interior. Life is too short for undocumented, proprietary data formats.<br>
<img width="16" height="16" src="cid:1__=0ABBFC16DFC7CEF18f9e8a93df938690@fws.gov" border="0" alt="Inactive hide details for "Rahkonen Jukka" <Jukka.Rahkonen@mmmtike.fi>">"Rahkonen Jukka" <Jukka.Rahkonen@mmmtike.fi><br>
<br>
<br>
<table width="100%" border="0" cellspacing="0" cellpadding="0">
<tr valign="top"><td style="background-image:url(cid:2__=0ABBFC16DFC7CEF18f9e8a93df938690@fws.gov); background-repeat: no-repeat; " width="40%">
<ul>
<ul>
<ul>
<ul><b><font size="2">"Rahkonen Jukka" <Jukka.Rahkonen@mmmtike.fi></font></b><font size="2"> </font>
<p><font size="2">12/07/2009 05:06 AM</font></ul>
</ul>
</ul>
</ul>
</td><td width="60%">
<table width="100%" border="0" cellspacing="0" cellpadding="0">
<tr valign="top"><td width="1%"><img width="58" height="1" src="cid:3__=0ABBFC16DFC7CEF18f9e8a93df938690@fws.gov" border="0" alt=""><br>
<div align="right"><font size="2">To</font></div></td><td width="100%"><img width="1" height="1" src="cid:3__=0ABBFC16DFC7CEF18f9e8a93df938690@fws.gov" border="0" alt=""><br>
<font size="2"><gdal-dev@lists.osgeo.org>, <Doug_Newcomb@fws.gov></font></td></tr>
<tr valign="top"><td width="1%"><img width="58" height="1" src="cid:3__=0ABBFC16DFC7CEF18f9e8a93df938690@fws.gov" border="0" alt=""><br>
<div align="right"><font size="2">cc</font></div></td><td width="100%"><img width="1" height="1" src="cid:3__=0ABBFC16DFC7CEF18f9e8a93df938690@fws.gov" border="0" alt=""><br>
</td></tr>
<tr valign="top"><td width="1%"><img width="58" height="1" src="cid:3__=0ABBFC16DFC7CEF18f9e8a93df938690@fws.gov" border="0" alt=""><br>
<div align="right"><font size="2">Subject</font></div></td><td width="100%"><img width="1" height="1" src="cid:3__=0ABBFC16DFC7CEF18f9e8a93df938690@fws.gov" border="0" alt=""><br>
<font size="2">Re: CUDA PyCUDA and GDAL</font></td></tr>
</table>
<table border="0" cellspacing="0" cellpadding="0">
<tr valign="top"><td width="58"><img width="1" height="1" src="cid:3__=0ABBFC16DFC7CEF18f9e8a93df938690@fws.gov" border="0" alt=""></td><td width="336"><img width="1" height="1" src="cid:3__=0ABBFC16DFC7CEF18f9e8a93df938690@fws.gov" border="0" alt=""></td></tr>
</table>
</td></tr>
</table>
<br>
<tt>> <Doug_Newcomb <at> fws.gov> writes:<br>
> <br>
> Hi Folks,Here's the gdal command (gdal 1.6.2) I used to merge ~3500 1<br>
meter NAIP<br>
> quarter quads (uncompressed geotiff TIFF) in 3 UTM projections into<br>
one Bigtiff Image<br>
> in the USGS Albers projection. It took about 15 hours ( on a 3 year<br>
old Intel Core2<br>
> Duo 64 bit Centos 5.3 Linux box with 6GB RAM) and created an<br>
uncompressed, tiled,<br>
> bigtiff file of 485 GB. About 32 GB/hr.<br>
> gdalwarp -t_srs "+proj=aea +lat_1=29.5+lat_2=45.5 +lat_0=23.0<br>
+lon_0=-96 <br>
> +x_0=0 +y_0=0 +ellps=GRS80 +datum=NAD83 +units=m no_defs <>"<br>
> -wo "SKIP_NOSOURCE" --config "GDAL_CACHEMAX=500" -wm=5000<br>
> -co "TILED=YES" */*.tif /biggis/albers/nc_naip2008.tif<br>
> <br>
> In the above command, -t_srs "+proj=aea +lat_1=29.5 +lat_2=45.5<br>
+lat_0=23.0 <br>
> +lon_0=-96 +x_0=0 +y_0=0 +ellps=GRS80 +datum=NAD83 +units=m no_defs<br>
<>"<br>
> indicates the target projection, -wo "SKIP_NOSOURCE" don't write<br>
> in areas for which there is no data for the current file, <br>
> --config "GDAL_CACHEMAX=500" set the cache memory at 500MB ( set this<br>
close <br>
> to the maximum input file size), -wm=5000 set the warp memory to<br>
5000MB ,<br>
> -co "TILED=YES" create a tiled tiff as output, */*.tif, use all of the<br>
tiffs <br>
> in all of the subdirectories as input files ( in this case there was<br>
one <br>
> directory for each of the 3 utm zones) ,<br>
/biggis/albers/nc_naip2008.tif,<br>
> the output file name and location. <br>
> <br>
<br>
Hi Doug,<br>
<br>
I finally tried your parameters and they did work fine for me also. I<br>
had something like hundred geotiffs, 400 MB each, and I was pushing<br>
them to bigtiff mosaic. I tried first with your *.tif selection and then<br>
again by using a virtual raster file as source, created from Mapserver<br>
tileindex shapefile with gdalbuildvrt. My Windows computer was handling<br>
about 20 GB/hour with cubic resampling (-rc) this time. Parameters <br>
-wo "SKIP_NOSOURCE" --config "GDAL_CACHEMAX=500" -wm=5000<br>
seem to have a big influence on efficiency. I wonder if there are some<br>
rules of thumb for selecting values of GDAL_CACHEMAX and -wm. You said<br>
cachemax is good to be close to the maximum input file size, how about<br>
-wm?<br>
<br>
-Jukka Rahkonen-<br>
</tt><br>
</body></html>