<div>Hi Markus,</div>
<div> </div>
<div>Many thanks for all your help. Thanks to your link to the wiki I've managed to get this 'poor man's paralellisation' up and running on our university's cluster and we've not had any problems thus far.<br>
<br></div>
<div class="gmail_quote">2008/12/26 Markus Neteler (via Nabble) <span dir="ltr"><<a href="http://n2.nabble.com/user/SendEmail.jtp?type=node&node=2092131&i=0" target="_top" rel="nofollow">ml-user%2B57828-1466323068@...</a>></span><br>
<blockquote class="gmail_quote" style="PADDING-LEFT: 1ex; MARGIN: 0px 0px 0px 0.8ex; BORDER-LEFT: #ccc 1px solid">Joseph, <br><br>I am using a cluster right now which is based on PBS to elaborate MODIS <br>satellite data. Some answers below: <br>
<br>On Feb 13, 2008 2:43 PM, joechip90 <<a href="http://n2.nabble.com/user/SendEmail.jtp?type=node&node=1881769&i=0" target="_blank" rel="nofollow">joechip90@...</a>> wrote: <br>> <br>> Dear All, <br>> <br>
> I have looked around on other postings and it appears that the majority (if <br>> not all) of the GRASS libraries are NOT thread safe. <br><br>Yes, unfortunately true. <br><br>> Unfortunately I have a <br>> very large processing job that would benefit from cluster processing. I <br>
> have written a script that can be run on multiple processors whilst being <br>> very careful not to allow different processes to try to modify the same data <br>> at any point. The same raster file is not accessed by different processes <br>
> at all in fact. <br><br>Yes, fine. Essentially there are at least two approaches of "poor man" <br>parallelization without modifying GRASS source code: <br><br>- split map into spatial chunks (possibly with overlap to gain smooth results) <br>
- time series: run each map elaboration on a different node. <br><br>> However, I also realise that alone might not solve all my problems. In any <br>> one process some temporary files are created (by GRASS libraries) and then <br>
> these are deleted on statup (cleaning temporary files...). Now I was <br>> wondering what these temporary files were and if there might be a problem <br>> with one process creating temporary files that it needs whilst another <br>
> process starts up GRASS and deletes them. Is there any way to call GRASS in <br>> a way that doesn't delete the temporary files? <br><br>You could just modify the start script and remove that call for "clean_temp". <br>
BUT: <br>I am currently elaborating some thousand maps for the same region (time <br>series). I elaborate each map in the same location but a different mapset <br>(simply using the map name as mapset name). At the end of the elaboration I <br>
call a second batch job which only contains g.copy to copy the result into a <br>common mapset. There is a low risk of race condition here in case that two <br>nodes finish at the same time but this could be even trapped in a loop which <br>
checks if the target mapset is locked and, if needed, launches g.copy again till <br>success. <br><br>> I appreciate that I'm trying to do something that GRASS doesn't really <br>> support but I was hoping that it might be possible to fiddle around and find <br>
> a way. Any help would be gratefully received. <br><br>To some extend GRASS supports what you need. <br>I have drafted a related wiki page at: <br><a href="http://grass.gdf-hannover.de/wiki/Parallel_GRASS_jobs" target="_blank" rel="nofollow">http://grass.gdf-hannover.de/wiki/Parallel_GRASS_jobs</a><br>
<br>Feel free to hack that page! <br><br>Good luck, <br>Markus <br>_______________________________________________ <br>grass-user mailing list <br><a href="http://n2.nabble.com/user/SendEmail.jtp?type=node&node=1881769&i=1" target="_blank" rel="nofollow">grass-user@...</a> <br>
<a href="http://lists.osgeo.org/mailman/listinfo/grass-user" target="_blank" rel="nofollow">http://lists.osgeo.org/mailman/listinfo/grass-user</a><br></blockquote></div><br>
<br><hr align="left" width="300">
View this message in context: <a href="http://n2.nabble.com/Multicore-Processing-and-Temporary-File-Cleanup-tp1881768p2092131.html">Re: Multicore Processing and Temporary File Cleanup</a><br>
Sent from the <a href="http://n2.nabble.com/Grass---Users-f1837860.html">Grass - Users mailing list archive</a> at Nabble.com.<br>