[GRASS-user] Multicore Processing and Temporary File Cleanup

[Markus Neteler]

On Feb 13, 2008 7:46 PM, Joseph Chipperfield <joechip90 at googlemail.com> wrote:
> Thank you Markus your Wiki entry is most helpful.
>
> It seems I need to make a few changes to my files and set up a large
> number of mapsets in every location.  Is it appropriate then to have
> multiple mapsets (one for each node) at a given location?

Sure!

> If so is
> there a way to automatically generate multiple mapsets in a given
> location such that I can jump straight into GRASS using a script command
> along the following in each of the processes (I will have thousands of
> processes)?

Yes. When you start GRASS with path to grassdada/location/mapset/ and
the mapset does not exist, it will be automatically created.
(hint for Hamish: this will then create a valid mapset, i.e. incl. DBF driver
 predefined - see grass-dev discussions)

As first step in your script, be sure to run
  g.mapsets add=mapset1_with_data[,mapset2_with_data]
to make the data to be elaborated accessible.

I am processing thousands of MODIS data like that right now:
- GRASS is launched as (example, indeed I loop over many map names)

loop over many map names, like "aqua_lst1km20020706.LST_Night_1km.filt"

------- snip -----------
MYMAPSET=$CURRMAP
TARGETMAPSET=results

GRASS_BATCH_JOB=/shareddisk/modis_job.sh
grass63 -text /shareddisk/grassdata/myloc/$MYMAPSET

# copy over result to target mapset
export INMAP=${CURRMAP}_rst
export INMAPSET=$MYMAPSET
export OUTMAP=$INMAP
export GRASS_BATCH_JOB=/shareddisk/gcopyjob.sh
grass63 -text  /shareddisk/grassdata/myloc/$TARGETMAPSET
exit 0
------- snap ----------

You see, that I run GRASS twice. Note that you need GRASS 6.3 to
make use of GRASS_BATCH_JOB (if present, GRASS automatically
executes that job instead of launching the user interface.

The script gcopyjob.sh simply contains
------- snip -----------
g.copy rast=$INMAP@$INMAPSET,$OUTMAP --o
------- snap ----------

That's it!

You script suggestion is essentially right. Only, you would better get
recent GRASS 6.3 to avoid a nightmare :)


> In each mapset would then contain the spatial data that each process
> will use.  You suggest then copying the output into a single shared
> mapset such as PERMANENT.  For my purposes I'll probably just save them
> as text files (the data then gets transferred to another program for the
> next stages of processing).

Sure - as you prefer. I put the elaborated MODIS map into a single mapset
for easy takeaway in the end.

I have extended
http://grass.gdf-hannover.de/wiki/Parallel_GRASS_jobs

Cheers
Markus

-- 
Markus Neteler
Fondazione Mach  -  Centre for Alpine Ecology
38100 Viote del Monte Bondone (Trento), Italy
neteler AT cealp.it      http://www.cealp.it/


> Again many thanks,
>
>