[GRASS-SVN] r43165 - grass-addons/tools/savanna
svn_grass at osgeo.org
svn_grass at osgeo.org
Thu Aug 19 11:57:34 EDT 2010
Author: jgrn307
Date: 2010-08-19 15:57:34 +0000 (Thu, 19 Aug 2010)
New Revision: 43165
Added:
grass-addons/tools/savanna/.project
grass-addons/tools/savanna/README
grass-addons/tools/savanna/savanna.sh
grass-addons/tools/savanna/savanna_copy_grass.sh
grass-addons/tools/savanna/savanna_copy_job.sh
grass-addons/tools/savanna/savanna_global_parameters.sh
grass-addons/tools/savanna/savanna_init_grass.sh
grass-addons/tools/savanna/savanna_init_job.sh
grass-addons/tools/savanna/savanna_mosaic_grass.sh
grass-addons/tools/savanna/savanna_mosaic_job.sh
grass-addons/tools/savanna/savanna_parameters_sample.sh
grass-addons/tools/savanna/savanna_tiling_grass.sh
grass-addons/tools/savanna/savanna_tiling_job.sh
Log:
Added: grass-addons/tools/savanna/.project
===================================================================
--- grass-addons/tools/savanna/.project (rev 0)
+++ grass-addons/tools/savanna/.project 2010-08-19 15:57:34 UTC (rev 43165)
@@ -0,0 +1,11 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<projectDescription>
+ <name>savanna</name>
+ <comment></comment>
+ <projects>
+ </projects>
+ <buildSpec>
+ </buildSpec>
+ <natures>
+ </natures>
+</projectDescription>
Property changes on: grass-addons/tools/savanna/.project
___________________________________________________________________
Added: svn:mime-type
+ text/plain
Added: grass-addons/tools/savanna/README
===================================================================
--- grass-addons/tools/savanna/README (rev 0)
+++ grass-addons/tools/savanna/README 2010-08-19 15:57:34 UTC (rev 43165)
@@ -0,0 +1,84 @@
+############################################################################
+#
+# MODULE: savanna
+# AUTHOR(S): Jonathan A. Greenberg <rgridengine at estarcion.net> and Quinn Hart
+# PURPOSE: Distributes GRASS raster commands across a gridengine cluster.
+# COPYRIGHT: (C) 2010 by the Jonathan A. Greenberg
+#
+# This program is free software under the GNU General Public
+# License (>=v2). Read the file COPYING that comes with GRASS
+# for details.
+#
+#############################################################################
+
+savanna is designed to distribute GRASS raster commands across a sun gridengine ("SGE") cluster.
+ The name "savanna" refers to the fact that we are creating a field of GRASS to solve our
+ raster problem.
+
+Overview:
+savanna takes a raster command and a region setting, and runs through these steps to distribute
+ the process across a gridengine cluster:
+1) Initialization: savanna creates a "grid" of polygons which represent the boundaries of each tile
+ which will be processed. This grid is based on the tile size, the overlap size, and the extent
+ of the g.region.
+2) Tiling: each tile is processed as a SGE job using GRASS's batch mode and g.regions set to the
+ tile extents.
+3) Copying: each output tile is then cropped (if there is an overlap) and copied back into a central directory.
+4) Mosaic: all of the output tiles are then mosaicked back into a single image.
+5) [TODO]: Once the process has been successfully run, the temporary tile mapsets/files are cleaned up.
+
+Usage:
+[COMING SOON]
+
+Configuring global parameters:
+Edit the "savanna_global_parameters.sh" file to fit your local installation.
+
+savanna_path: the full path to the savanna*.sh files (where you are probably reading this README)
+savanna_tmp: a path to the temporary outputs of savanna (this is where the output and error logfiles
+ are stored of the run, so its a good place to check to see how the process is proceding and for
+ troubleshooting.
+gridengine_settings: this is the full path to the gridengine "settings.sh" file. Your local admin
+ should know where this is, if gridengine is installed and running. A standard install would be
+ something like "/pathto/ge6.2u5/default/common/settings.sh"
+
+Configuring savanna parameter file:
+[this will be a command line argument via getopts eventually, right now you have to edit savanna.sh to the
+ parameter file.]
+Important: DON'T FORGET THE @mapset -- this scripts launches externally to GRASS, so it has no idea
+ what mapset you want to use.
+Note the use of \ if you need to continue to the next line.
+
+grassdata_directory: the full path to your grass datra directory, e.g.:
+ grassdata_directory='/pathto/grass'
+base_location: the location the savanna run will be performed in, e.g.:
+ base_location='savanna_location'
+gregion: this needs to be a properly formatted g.region statement using rast=raster at mapset, example:
+ "g.region rast=savanna_test at savanna_test_mapset"
+command: a properly formatted r.* command that outputs raster files of the same extent as the input, example:
+ "r.slope.aspect elevation=savanna_test at savanna_test_mapset \
+ slope=savanna_test_slope \
+ aspect=savanna_test_aspect \
+ format=degrees prec=float zfactor=1.0 min_slp_allowed=0.0"
+outputs: this needs to be the output of the raster statement; use wildcards if need be. Without this, savanna
+ can't perform the copy and mosaic phases (it has no idea what the outputs of your command are). Example:
+ outputs="savanna_test_*"
+tilexsize and tileysize: these are the requested tile sizes per gridengine job. The idea is to get these tiles small
+ enough to keep the memory footprint low, while not excessively small so that there is a lot of overhead during
+ the copy/mosaic phases. Example:
+ tilexsize=500
+ tileysize=500
+tileoverlap: if your process requires some level of local neighborhood (e.g. slope and aspect), you need to set an overlap
+ between tiles, otherwise you will have seamlines resulting. Set this depending on your particular process (straight
+ raster algebra via mapcalc, for instance, could have the overlap set to 0.
+ tileoverlap=10
+mask: UNUSED RIGHT NOW
+
+TODO:
+Lots of error checking.
+Better command line interface.
+Include example raster.
+Cleanup.
+
+LINKS:
+SGE setup
+SGE main site
\ No newline at end of file
Added: grass-addons/tools/savanna/savanna.sh
===================================================================
--- grass-addons/tools/savanna/savanna.sh (rev 0)
+++ grass-addons/tools/savanna/savanna.sh 2010-08-19 15:57:34 UTC (rev 43165)
@@ -0,0 +1,183 @@
+#!/bin/bash
+#$ -S /bin/bash
+############################################################################
+#
+# MODULE: savanna
+# AUTHOR(S): Jonathan A. Greenberg <savanna at estarcion.net> and Quinn Hart
+# PURPOSE: Distributes R raster commands across a gridengine cluster.
+# COPYRIGHT: (C) 2010 by the Jonathan A. Greenberg
+#
+# This program is free software under the GNU General Public
+# License (>=v2). Read the file COPYING that comes with GRASS
+# for details.
+#
+#############################################################################
+
+echo 'Setting up parameters...'
+. "savanna_global_parameters.sh"
+export parameter_file="/home/jongreen/code/grass/savanna/v06/savanna_parameters_00008m_rsun.sh"
+. $parameter_file
+
+# These should be left alone without good reason.
+export savanna_init_job_file="savanna_init_job.sh"
+export savanna_init_grass_file="savanna_init_grass.sh"
+export savanna_tiling_job_file="savanna_tiling_job.sh"
+export savanna_tiling_grass_file="savanna_tiling_grass.sh"
+export savanna_copy_job_file="savanna_copy_job.sh"
+export savanna_copy_grass_file="savanna_copy_grass.sh"
+export savanna_mosaic_job_file="savanna_mosaic_job.sh"
+export savanna_mosaic_grass_file="savanna_mosaic_grass.sh"
+
+echo 'Checking gridengine...'
+. $gridengine_settings
+
+# Check for all required files.
+
+# Create a unique name based on time used as the prefix.
+unique_prefix_suffix=$(date +%s)
+export prefix="savanna_"$unique_prefix_suffix
+
+# Some standard naming for temp files.
+export tiling_grid_suffix="_tiling_grid"
+
+# Create working directory.
+# Should use mktemp -d
+export working_tmp=$savanna_tmp/$prefix
+mkdir $working_tmp
+
+### Step 1: Initialize tiling.
+echo 'Creating tiling script...'
+
+# export init_prefix=$prefix"_init"
+export init_mapset=$prefix"_init"
+
+# echo $working_tmp/areas.txt
+
+echo "qsub -N ${prefix}_init -sync y -o $working_tmp -e $working_tmp \
+-v \
+savanna_path=$savanna_path,\
+savanna_init_job_file=$savanna_init_job_file,\
+savanna_init_grass_file=$savanna_init_grass_file,\
+tilexsize=$tilexsize,\
+tileysize=$tileysize,\
+tileoverlap=$tileoverlap,\
+grassdata_directory=$grassdata_directory,\
+base_location=$base_location,\
+init_mapset=$init_mapset,\
+tiling_grid=$init_mapset$tiling_grid_suffix,\
+gregion=\"$gregion\",\
+mask=$mask,\
+areas_file=$working_tmp/areas.txt \
+$savanna_path/$savanna_init_job_file" | sort > $working_tmp/${prefix}_sge_init.sh
+
+# Submit the initialization script to grid engine.
+echo 'Submitting tiling script to gridengine...'
+sh $working_tmp/${prefix}_sge_init.sh
+
+### Step 2: Perform tiled processing.
+# Read in number of areas.
+. $working_tmp/areas.txt
+# areas=108
+numdigits=$(expr length $areas)
+
+for tile in $(seq 1 $areas)
+do
+tile_unique_id=$(printf "%0"$numdigits"d\n" $tile)
+echo "qsub -N ${prefix}_tiling_$tile_unique_id -o $working_tmp -e $working_tmp \
+-hold_jid ${prefix}_init \
+-v \
+tile=$tile,\
+savanna_path=$savanna_path,\
+savanna_tiling_job_file=$savanna_tiling_job_file,\
+savanna_tiling_grass_file=$savanna_tiling_grass_file,\
+tileoverlap=$tileoverlap,\
+grassdata_directory=$grassdata_directory,\
+base_location=$base_location,\
+init_mapset=$init_mapset,\
+tile_mapset=$prefix\_$tile_unique_id,\
+tiling_grid=$init_mapset$tiling_grid_suffix,\
+additional_searchpaths=$additional_searchpaths,\
+gregion=\"$gregion\",\
+command=\"$command\" \
+$savanna_path/$savanna_tiling_job_file"
+done > $working_tmp"/"$prefix"_sge_tiling.sh"
+
+sh $working_tmp"/"$prefix"_sge_tiling.sh"
+
+### Step 3: Crop and copy the tiles into a single mapset
+# Note: this seems to interrupt itself with concurrent mapset access, so
+# I am going to restrict it based on the previous tile.
+
+# The first tile has no dependency on other tiles, so we allow it to run as usual:
+
+let tile=1
+tile_unique_id=$(printf "%0"$numdigits"d\n" $tile)
+echo "qsub -N ${prefix}_copy_$tile_unique_id -o $working_tmp -e $working_tmp \
+-v \
+tile=$tile,\
+tile_unique_id=$tile_unique_id,\
+prefix=$prefix,\
+savanna_path=$savanna_path,\
+savanna_copy_job_file=$savanna_copy_job_file,\
+savanna_copy_grass_file=$savanna_copy_grass_file,\
+grassdata_directory=$grassdata_directory,\
+base_location=$base_location,\
+init_mapset=$init_mapset,\
+tile_mapset=$prefix\_$tile_unique_id,\
+tiling_grid=$init_mapset$tiling_grid_suffix,\
+outputs=\"${outputs[*]}\",\
+gregion=\"$gregion\" \
+$savanna_path/$savanna_copy_job_file" > $working_tmp"/"$prefix"_sge_copy.sh"
+
+for tile in $(seq 2 $areas)
+do
+let tile_prev=$tile-1
+tile_unique_id_prev=$(printf "%0"$numdigits"d\n" $tile_prev)
+tile_unique_id=$(printf "%0"$numdigits"d\n" $tile)
+echo "qsub -N ${prefix}_copy_$tile_unique_id -o $working_tmp -e $working_tmp \
+-hold_jid ${prefix}_copy_$tile_unique_id_prev \
+-v \
+tile=$tile,\
+tile_unique_id=$tile_unique_id,\
+prefix=$prefix,\
+savanna_path=$savanna_path,\
+savanna_copy_job_file=$savanna_copy_job_file,\
+savanna_copy_grass_file=$savanna_copy_grass_file,\
+grassdata_directory=$grassdata_directory,\
+base_location=$base_location,\
+init_mapset=$init_mapset,\
+tile_mapset=$prefix\_$tile_unique_id,\
+tiling_grid=$init_mapset$tiling_grid_suffix,\
+outputs=\"${outputs[*]}\",\
+gregion=\"$gregion\" \
+$savanna_path/$savanna_copy_job_file"
+done >> $working_tmp"/"$prefix"_sge_copy.sh"
+
+sh $working_tmp"/"$prefix"_sge_copy.sh"
+
+
+### Step 4: Mosaic all the files together.
+# Note: this seems to interrupt itself with concurrent mapset access, so
+# I am going to restrict it based on the previous tile.
+
+echo qsub -N ${prefix}_mosaic -o $working_tmp -e $working_tmp \
+-hold_jid ${prefix}_copy_"*" \
+-v \
+savanna_path=$savanna_path,\
+savanna_mosaic_job_file=$savanna_mosaic_job_file,\
+savanna_mosaic_grass_file=$savanna_mosaic_grass_file,\
+grassdata_directory=$grassdata_directory,\
+base_location=$base_location,\
+init_mapset=$init_mapset,\
+prefix=$prefix,\
+tiling_grid=$init_mapset$tiling_grid_suffix,\
+gregion=\"$gregion\",\
+outputs=\"${outputs[*]}\",\
+numdigits=$numdigits,\
+areas_file=$working_tmp/areas.txt \
+$savanna_path/$savanna_mosaic_job_file | sort > $working_tmp/$prefix\_sge\_mosaic.sh
+
+echo 'Submitting mosaic script to gridengine...'
+sh $working_tmp/$prefix\_sge\_mosaic.sh
+
+exit 0
Added: grass-addons/tools/savanna/savanna_copy_grass.sh
===================================================================
--- grass-addons/tools/savanna/savanna_copy_grass.sh (rev 0)
+++ grass-addons/tools/savanna/savanna_copy_grass.sh 2010-08-19 15:57:34 UTC (rev 43165)
@@ -0,0 +1,80 @@
+#!/bin/bash
+#$ -S /bin/bash
+############################################################################
+#
+# MODULE: savanna
+# AUTHOR(S): Jonathan A. Greenberg <savanna at estarcion.net> and Quinn Hart
+# PURPOSE: Distributes R raster commands across a gridengine cluster.
+# COPYRIGHT: (C) 2010 by the Jonathan A. Greenberg
+#
+# This program is free software under the GNU General Public
+# License (>=v2). Read the file COPYING that comes with GRASS
+# for details.
+#
+#############################################################################
+# This is a single r.sun job for use with grid engine. The inputs are:
+# grassdata_directory = base grass data directory
+# base_location = grass location
+# base_mapset = grass mapset
+# output_prefix
+# output_suffix
+# dem =
+# tiling_vector = vector containing rectangular polygons defining each tile
+# overlapsize = buffering size in pixels to use for each tile
+# tile = tile ID
+# TODO: This can probably be parallized, if we put a wait condition for any other similarly named jobs.
+
+outputs=($outputs)
+
+g.region -d
+$gregion
+g.region -p
+if [ $? -ne 0 ] ; then
+ echo 'Error in g.region pass 1...'
+ exit 1 #Abandon the loop.
+fi
+
+# tile_unique_id=$(printf "%0"$numdigits"d\n" $tile)
+
+g.region -g vect=${tiling_grid}_sub_temp@${prefix}_$tile_unique_id
+if [ $? -ne 0 ] ; then
+ echo 'Error in g.region pass 2 for tile '$tile'...'
+ exit 1 #Abandon the loop.
+fi
+g.region -p
+eval `g.region -g`
+# I don't know why, but the eastern edge is one half off, so we must correct it.
+# corr_e=$(echo "scale=0; ($e-($ewres*0.5))" | bc)
+
+# g.region -a vect=${tiling_grid}_sub_temp@${prefix}_$tile_unique_id e=$corr_e
+# g.region -p
+# if [ $? -ne 0 ] ; then
+# echo 'Error in g.region pass 4 for tile '$tile'...'
+# exit 1 #Abandon the loop.
+# fi
+
+# We loop through each of the outputs
+outputs_N=${#outputs[*]}
+# echo $outputs_N
+unset MAPS
+for output_loop in $(seq 1 $outputs_N)
+do
+# echo $output_loop
+# echo "g.mlist type=rast sep=space mapset=${prefix}_$tile_unique_id pat=${outputs[($output_loop-1)]}"
+# echo "MAPS=(`g.mlist type=rast sep=space mapset=${prefix}_$tile_unique_id pat=${outputs[($output_loop-1)]}`)"
+MAPS=(`g.mlist type=rast sep=space mapset=${prefix}_$tile_unique_id pat=${outputs[($output_loop-1)]}`)
+MAPS_N=${#MAPS[*]}
+for maps_loop in $(seq 1 $MAPS_N)
+do
+echo $maps_loop
+g.region -p
+r.mapcalc ${MAPS[($maps_loop-1)]}_tile_$tile_unique_id=${MAPS[($maps_loop-1)]}@${prefix}_$tile_unique_id
+if [ $? -ne 0 ] ; then
+ echo 'Error in r.mapcalc for '$tile'...'
+ exit 1 #Abandon the loop.
+fi
+done
+unset MAPS
+done
+
+exit 0
Property changes on: grass-addons/tools/savanna/savanna_copy_grass.sh
___________________________________________________________________
Added: svn:executable
+ *
Added: grass-addons/tools/savanna/savanna_copy_job.sh
===================================================================
--- grass-addons/tools/savanna/savanna_copy_job.sh (rev 0)
+++ grass-addons/tools/savanna/savanna_copy_job.sh 2010-08-19 15:57:34 UTC (rev 43165)
@@ -0,0 +1,31 @@
+#!/bin/bash
+#$ -S /bin/bash
+############################################################################
+#
+# MODULE: savanna
+# AUTHOR(S): Jonathan A. Greenberg <savanna at estarcion.net> and Quinn Hart
+# PURPOSE: Distributes R raster commands across a gridengine cluster.
+# COPYRIGHT: (C) 2010 by the Jonathan A. Greenberg
+#
+# This program is free software under the GNU General Public
+# License (>=v2). Read the file COPYING that comes with GRASS
+# for details.
+#
+##############################################################################!/bin/bash
+#$ -S /bin/bash
+
+# This initializes a single grass tile via gridengine.
+
+# Note: this sets up an infinite execution loop -- we should probably give
+# 10 tries or something and bail.
+
+export GRASS_BATCH_JOB=$savanna_path/$savanna_copy_grass_file
+#while true; do
+grass64 -c -text $grassdata_directory/$base_location/$init_mapset
+ if [ $? -eq 0 ] ; then
+ exit 1 #Abandon the loop.
+ else
+#sleep 30
+#done
+ exit 0
+ fi
Added: grass-addons/tools/savanna/savanna_global_parameters.sh
===================================================================
--- grass-addons/tools/savanna/savanna_global_parameters.sh (rev 0)
+++ grass-addons/tools/savanna/savanna_global_parameters.sh 2010-08-19 15:57:34 UTC (rev 43165)
@@ -0,0 +1,14 @@
+# Set these based on your local environment distribution.
+export savanna_path="/pathto/savanna"
+export savanna_tmp="/tmp/savanna"
+export gridengine_settings="/pathto/ge6.2u5/default/common/settings.sh"
+
+# These should be left alone without good reason.
+export savanna_init_job_file="savanna_init_job.sh"
+export savanna_init_grass_file="savanna_init_grass.sh"
+export savanna_tiling_job_file="savanna_tiling_job.sh"
+export savanna_tiling_grass_file="savanna_tiling_grass.sh"
+export savanna_copy_job_file="savanna_copy_job.sh"
+export savanna_copy_grass_file="savanna_copy_grass.sh"
+export savanna_mosaic_job_file="savanna_mosaic_job.sh"
+export savanna_mosaic_grass_file="savanna_mosaic_grass.sh"
\ No newline at end of file
Added: grass-addons/tools/savanna/savanna_init_grass.sh
===================================================================
--- grass-addons/tools/savanna/savanna_init_grass.sh (rev 0)
+++ grass-addons/tools/savanna/savanna_init_grass.sh 2010-08-19 15:57:34 UTC (rev 43165)
@@ -0,0 +1,87 @@
+#!/bin/bash
+#$ -S /bin/bash
+############################################################################
+#
+# MODULE: savanna
+# AUTHOR(S): Jonathan A. Greenberg <savanna at estarcion.net> and Quinn Hart
+# PURPOSE: Distributes R raster commands across a gridengine cluster.
+# COPYRIGHT: (C) 2010 by the Jonathan A. Greenberg
+#
+# This program is free software under the GNU General Public
+# License (>=v2). Read the file COPYING that comes with GRASS
+# for details.
+#
+##############################################################################!/bin/bash
+
+# savanna GRASS_BATCH_JOB to create the tiling grid.
+
+# Set the gregion
+$gregion
+eval `$gregion -g`
+# g.region -p
+
+if [ $? -ne 0 ] ; then
+ echo 'Error in setting g.region...'
+ exit 1 #Abandon the loop.
+fi
+
+# Calculate the base number of tiles to use.
+numxtiles=$(echo "scale=0; (($w - $e)/$ewres)/$tilexsize" | bc)
+if ((numxtiles < 0)) ; then
+ let "numxtiles = numxtiles * -1"
+fi
+smallxtilesize=$(echo "scale=0; (($w - $e)/$ewres) % $tilexsize" | bc)
+if ((smallxtilesize != 0)); then
+ let 'numxtiles++'
+fi
+
+numytiles=$(echo "scale=0; (($n - $s)/$nsres)/$tileysize" | bc)
+if ((numytiles < 0))
+ then let "numytiles = numytiles * -1"
+fi
+smallytilesize=$(echo "scale=0; (($n - $s)/$nsres) % $tileysize" | bc)
+if ((smallytilesize != 0))
+ then let 'numytiles++'
+fi
+
+eval $gregion
+let 'boxx = ewres*tilexsize'
+let 'boxy = nsres*tileysize'
+
+v.mkgrid map=$tiling_grid grid=$numytiles,$numxtiles position=coor coor=$w,$s box=$boxy,$boxx angle=0 --overwrite --quiet
+if [ $? -ne 0 ] ; then
+ echo 'Error in v.mkgrid...'
+ exit 1 #Abandon the loop.
+fi
+
+# The next step should be optional only if the user provided a mask.
+#v.select ainput=$output_prefix$tiling_grid_suffix atype=area alayer=1 binput=$mask btype=point,line,area blayer=1 output=$tiling_vector"_uncor" operator=overlap --overwrite --quiet
+#if [ $? -ne 0 ] ; then
+# echo 'Error in v.select...'
+# exit 1 #Abandon the loop.
+#fi
+#v.category input=$tiling_vector"_uncor" output=$tiling_vector"_stripped" option=del --overwrite --quiet
+#if [ $? -ne 0 ] ; then
+# echo 'Error in v.category pass 1...'
+# exit 1 #Abandon the loop.
+#fi
+#v.category input=$tiling_vector"_stripped" output=$tiling_vector option=add type=centroid layer=1 cat=1 step=1 --overwrite --quiet
+#if [ $? -ne 0 ] ; then
+# echo 'Error in v.category pass 2...'
+# exit 1 #Abandon the loop.
+#fi
+
+# Need to do some cleaning
+
+eval `v.info -t map=$tiling_grid layer=1 --quiet`
+if [ $? -ne 0 ] ; then
+ echo 'Error in v.info...'
+ exit 1 #Abandon the loop.
+fi
+
+echo $areas
+echo $areas_file
+
+echo export areas=$areas > $areas_file
+
+exit 0
Property changes on: grass-addons/tools/savanna/savanna_init_grass.sh
___________________________________________________________________
Added: svn:executable
+ *
Added: grass-addons/tools/savanna/savanna_init_job.sh
===================================================================
--- grass-addons/tools/savanna/savanna_init_job.sh (rev 0)
+++ grass-addons/tools/savanna/savanna_init_job.sh 2010-08-19 15:57:34 UTC (rev 43165)
@@ -0,0 +1,29 @@
+#!/bin/bash
+#$ -S /bin/bash
+############################################################################
+#
+# MODULE: savanna
+# AUTHOR(S): Jonathan A. Greenberg <savanna at estarcion.net> and Quinn Hart
+# PURPOSE: Distributes R raster commands across a gridengine cluster.
+# COPYRIGHT: (C) 2010 by the Jonathan A. Greenberg
+#
+# This program is free software under the GNU General Public
+# License (>=v2). Read the file COPYING that comes with GRASS
+# for details.
+#
+##############################################################################!/bin/bash
+#$ -S /bin/bash
+
+# Note: this sets up an infinite execution loop -- we should probably give
+# 10 tries or something and bail.
+
+export GRASS_BATCH_JOB=$savanna_path/$savanna_init_grass_file
+echo $GRASS_BATCH_JOB
+# while true; do
+grass64 -c -text $grassdata_directory/$base_location/$init_mapset
+# if [ $? -eq 0 ] ; then
+# exit 1 #Abandon the loop.
+# fi
+# sleep 1
+# done
+
Added: grass-addons/tools/savanna/savanna_mosaic_grass.sh
===================================================================
--- grass-addons/tools/savanna/savanna_mosaic_grass.sh (rev 0)
+++ grass-addons/tools/savanna/savanna_mosaic_grass.sh 2010-08-19 15:57:34 UTC (rev 43165)
@@ -0,0 +1,73 @@
+#!/bin/bash
+#$ -S /bin/bash
+############################################################################
+#
+# MODULE: savanna
+# AUTHOR(S): Jonathan A. Greenberg <savanna at estarcion.net> and Quinn Hart
+# PURPOSE: Distributes R raster commands across a gridengine cluster.
+# COPYRIGHT: (C) 2010 by the Jonathan A. Greenberg
+#
+# This program is free software under the GNU General Public
+# License (>=v2). Read the file COPYING that comes with GRASS
+# for details.
+#
+##############################################################################!/bin/bash
+#$ -S /bin/bash
+# This is a single r.sun job for use with grid engine. The inputs are:
+# grassdata_directory = base grass data directory
+# base_location = grass location
+# base_mapset = grass mapset
+# output_prefix
+# output_suffix
+# dem =
+# tiling_vector = vector containing rectangular polygons defining each tile
+# overlapsize = buffering size in pixels to use for each tile
+# tile = tile ID
+# TODO: This can probably be parallized, if we put a wait condition for any other similarly named jobs.
+
+outputs=($outputs)
+
+$gregion
+if [ $? -ne 0 ] ; then
+ echo 'Error in g.region pass 1...'
+ exit 1 #Abandon the loop.
+fi
+
+outputs_N=${#outputs[*]}
+# We use the first tile to define what the outputs are.
+tile_unique_id=$(printf "%0"$numdigits"d\n" 1)
+unset MAPS
+for output_loop in $(seq 1 $outputs_N)
+do
+echo $output_loop
+echo "base_MAPS=(g.mlist type=rast sep=space mapset=${prefix}_$tile_unique_id pat=${outputs[($output_loop-1)]})"
+base_MAPS=(`g.mlist type=rast sep=space mapset=${prefix}_$tile_unique_id pat=${outputs[($output_loop-1)]}`)
+base_MAPS_N=${#base_MAPS[*]}
+echo $base_MAPS
+for maps_loop in $(seq 1 $base_MAPS_N)
+do
+echo $maps_loop
+echo "MAPS=(`g.mlist type=rast sep=, pat=${base_MAPS[($maps_loop-1)]}_tile_*`)"
+MAPS=(`g.mlist type=rast sep=, pat=${base_MAPS[($maps_loop-1)]}_tile_*`)
+echo $MAPS
+g.region rast=${MAPS[*]}
+if [ $? -ne 0 ] ; then
+ echo 'Error in g.region pass 5 for beam...'
+ exit 1 #Abandon the loop.
+fi
+r.patch in=$MAPS out=${base_MAPS[($maps_loop-1)]}
+#if [ $? -ne 0 ] ; then
+# echo 'Error in r.patch...'
+# exit 1 #Abandon the loop.
+#fi
+unset MAPS
+done
+unset base_MAPS
+done
+
+# Now clean up the files
+# g.mremove -f rast=$tile_output_name"_*_beam",$tile_output_name"_*_diff"
+# echo $grassdata_directory/$base_location/$tile_output_name"_*"
+# rm -rf $grassdata_directory/$base_location/$tile_output_name"_*"
+
+exit 0
Property changes on: grass-addons/tools/savanna/savanna_mosaic_grass.sh
___________________________________________________________________
Added: svn:executable
+ *
Added: grass-addons/tools/savanna/savanna_mosaic_job.sh
===================================================================
--- grass-addons/tools/savanna/savanna_mosaic_job.sh (rev 0)
+++ grass-addons/tools/savanna/savanna_mosaic_job.sh 2010-08-19 15:57:34 UTC (rev 43165)
@@ -0,0 +1,37 @@
+#!/bin/bash
+#$ -S /bin/bash
+############################################################################
+#
+# MODULE: savanna
+# AUTHOR(S): Jonathan A. Greenberg <savanna at estarcion.net> and Quinn Hart
+# PURPOSE: Distributes R raster commands across a gridengine cluster.
+# COPYRIGHT: (C) 2010 by the Jonathan A. Greenberg
+#
+# This program is free software under the GNU General Public
+# License (>=v2). Read the file COPYING that comes with GRASS
+# for details.
+#
+##############################################################################!/bin/bash
+#$ -S /bin/bash
+
+export tile_output_name=$output_prefix"_"$day_unique_id
+# export temp_mapset=$tile_output_name
+# This could set up an infinite loops, so be careful.
+# We need to surpress the output -- this loop works fine
+# except the output file will grow out of control.
+export GRASS_BATCH_JOB=$savanna_path/$savanna_mosaic_grass_file
+# for i in (( ; ; )); do
+# while true; do
+grass64 -c -text $grassdata_directory/$base_location/$init_mapset
+# if [ $? -eq 0 ] ; then
+# exit 1 #Abandon the loop.
+# fi
+# sleep 1
+# done
+# Once its done, remove the temp directories (BE CAREFUL WITH THIS).
+
+# eval rm -rf $grassdata_directory'/'$base_location'/'$tile_output_name'*'
+# ls $grassdata_directory'/'$base_location'/'$tile_output_name'*'
+# echo $grassdata_directory"/"$base_location"/"$tile_output_name"_*"
+
+
Added: grass-addons/tools/savanna/savanna_parameters_sample.sh
===================================================================
--- grass-addons/tools/savanna/savanna_parameters_sample.sh (rev 0)
+++ grass-addons/tools/savanna/savanna_parameters_sample.sh 2010-08-19 15:57:34 UTC (rev 43165)
@@ -0,0 +1,15 @@
+export grassdata_directory="/opt/comet/downscale/grass"
+export base_location="california_albers"
+export gregion="g.region rast=savanna_test at savanna_test"
+export command="r.slope.aspect elevation=ned_ca_masked_epsg3310 at ned \
+slope=ned_ca_masked_epsg3310_comet_00008m_slope \
+aspect=ned_ca_masked_epsg3310_comet_00008m_aspect \
+format=degrees prec=float zfactor=1.0 min_slp_allowed=0.0"
+# outputs should be a list or prefix
+unset outputs
+export outputs="ned_ca_masked_epsg3310_comet_00008m_*"
+export tilexsize="500"
+export tileysize="500"
+export tileoverlap="10"
+export mask="ned_ca_masked_epsg3310_comet_00008m"
+
Property changes on: grass-addons/tools/savanna/savanna_parameters_sample.sh
___________________________________________________________________
Added: svn:executable
+ *
Added: grass-addons/tools/savanna/savanna_tiling_grass.sh
===================================================================
--- grass-addons/tools/savanna/savanna_tiling_grass.sh (rev 0)
+++ grass-addons/tools/savanna/savanna_tiling_grass.sh 2010-08-19 15:57:34 UTC (rev 43165)
@@ -0,0 +1,61 @@
+#!/bin/bash
+#$ -S /bin/bash
+############################################################################
+#
+# MODULE: savanna
+# AUTHOR(S): Jonathan A. Greenberg <savanna at estarcion.net> and Quinn Hart
+# PURPOSE: Distributes R raster commands across a gridengine cluster.
+# COPYRIGHT: (C) 2010 by the Jonathan A. Greenberg
+#
+# This program is free software under the GNU General Public
+# License (>=v2). Read the file COPYING that comes with GRASS
+# for details.
+#
+##############################################################################!/bin/bash
+
+# savanna GRASS_BATCH_JOB to process a single tile.
+
+# Set the gregion
+$gregion
+ if [ $? -ne 0 ] ; then
+ echo 'Error in setting g.region...'
+ exit 1 #Abandon the loop.
+ fi
+
+eval `$gregion -g`
+
+# Add additional search paths, if neccessary.
+g.mapsets addmapset=$additional_searchpaths 'fs= ,'
+
+g.region -p
+# echo $tiling_grid"@"$init_mapset
+
+ let "buffersize=ewres*tileoverlap"
+
+ v.extract input=$tiling_grid"@"$init_mapset \
+ output=$tiling_grid"_sub_temp" type=area layer=1 new=-1 list=$tile --overwrite
+ if [ $? -ne 0 ] ; then
+ echo 'Error in v.extract...'
+ exit 1 #Abandon the loop.
+ fi
+ v.buffer input=$tiling_grid"_sub_temp" \
+ output=$tiling_grid"_sub_temp_buffer" \
+ type=area layer=1 buffer=$buffersize scale=1.0 tolerance=0.01 --overwrite --quiet
+ if [ $? -ne 0 ] ; then
+ echo 'Error in v.buffer...'
+ exit 1 #Abandon the loop.
+ fi
+ g.region -a vect=$tiling_grid"_sub_temp_buffer" --quiet
+ g.region -p
+ if [ $? -ne 0 ] ; then
+ echo 'Error in setting g.region...'
+ exit 1 #Abandon the loop.
+ fi
+echo $command
+$command
+ if [ $? -ne 0 ] ; then
+ echo 'Error in command...'
+ exit 1 #Abandon the loop.
+ fi
+
+exit 0
Property changes on: grass-addons/tools/savanna/savanna_tiling_grass.sh
___________________________________________________________________
Added: svn:executable
+ *
Added: grass-addons/tools/savanna/savanna_tiling_job.sh
===================================================================
--- grass-addons/tools/savanna/savanna_tiling_job.sh (rev 0)
+++ grass-addons/tools/savanna/savanna_tiling_job.sh 2010-08-19 15:57:34 UTC (rev 43165)
@@ -0,0 +1,30 @@
+#!/bin/bash
+#$ -S /bin/bash
+############################################################################
+#
+# MODULE: savanna
+# AUTHOR(S): Jonathan A. Greenberg <savanna at estarcion.net> and Quinn Hart
+# PURPOSE: Distributes R raster commands across a gridengine cluster.
+# COPYRIGHT: (C) 2010 by the Jonathan A. Greenberg
+#
+# This program is free software under the GNU General Public
+# License (>=v2). Read the file COPYING that comes with GRASS
+# for details.
+#
+##############################################################################!/bin/bash
+#$ -S /bin/bash
+
+# This initializes a single grass tile via gridengine.
+
+# Note: this sets up an infinite execution loop -- we should probably give
+# 10 tries or something and bail.
+
+export GRASS_BATCH_JOB=$savanna_path/$savanna_tiling_grass_file
+# while true; do
+grass64 -c -text $grassdata_directory/$base_location/$tile_mapset
+ if [ $? -eq 0 ] ; then
+ exit 1 #Abandon the loop.
+ fi
+# sleep 1
+# done
+
More information about the grass-commit
mailing list