Hi Daniel,<br><br><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Let's say you have three different tasks that need to be performed for each map. This is just a short example, you'd need to adapt it further, but you could do something like this:<br>
<br>running_jobs = 0<div>
<br>
# Loop over jobs<br>for i in range(jobs):<br></div> # Check what position you're on in your array of jobs<br> position = i % 3<br> # Do a job accordingly<br> if position == 0:<br>
do_something()<br>
elif position == 1:<br>
do_something_else()<br>
elif position == 2:<br>
and_yet_something_else()<br>
# Increase count of running jobs<br> running_jobs += 1<br>
# If you don't have any available workers, wait until they're all done<br>
if running_jobs % workers is 0:<br>
for j in range(workers):<br>
wait_for_jobs_to_finish()<br>
# Now reset the running jobs to 0 so that you can continue to add jobs to your queue<br>
running_jobs = 0<br><br></blockquote><div><br>In your loop what do you mean with "job" respectivle the number of jobs. Is that e.g. 3 task x 200 maps = 600 jobs?<br><br>Attached there is a little python script which applies a focal filter on a raster map but based on a user supplied r.cost<br>
distance from each cell instead of a moving window. Thus the script loops over each single raster cell (transfered to a <br>temporary point vector) and calculates r. cost, mapcalc and r.univar which takes quite a lot of time. The script is working<br>
so far and uses g.parser to generate the gui interface etc.<br><br>I think I can upload this skript to the addons/wiki but before I'd like to make it faster (e.g. parallize the code).<br><br>Thank you for your help so far...<br>
<br>/Johannes<br><br><br><br> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">HTH!<div><div><br>Daniel<br>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px"><span style="font-size:13px;font-family:arial,sans-serif"></span></p><p>--<br></p><p>B.Sc. Daniel Lee<br>
Geschäftsführung für Forschung und Entwicklung<br>ISIS - International Solar Information Solutions GbR<br>Vertreten durch: Daniel Lee, Nepomuk Reinhard und Nils Räder<br></p><p>Softwarecenter 3<br>35037 Marburg<br>Festnetz: <a value="+4964213796256" style="color:rgb(28,81,168)">+49 6421 379 6256</a><br>
Mobil: <a value="+4917661277269" style="color:rgb(28,81,168)">+49 176 6127 7269</a><br>E-Mail: <a href="mailto:Lee@isi-solutions.org" style="color:rgb(28,81,168)" target="_blank">Lee@isi-solutions.org</a><br>Web: <a href="http://www.isi-solutions.org/" style="color:rgb(28,81,168)" target="_blank">http://www.isi-solutions.org</a></p>
<p></p><br>
<br><br><div class="gmail_quote">2012/8/7 Johannes Radinger <span dir="ltr"><<a href="mailto:johannesradinger@gmail.com" target="_blank">johannesradinger@gmail.com</a>></span><br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Hi,<br><br>sounds promising, but somehow I don't get it (as I not yet deeply into python scripts :()<br>E.g if I try to perform a second step using the output of the first step (r.slope.aspect) as<br>input in the next one (e.g. r.cost) ... how would you do that. I understand in you example that<br>
you use the modulus operator to query if the last job of each "group" is started...then the wait<br>is used to finish all jobs of that group before going to the next line. If I understand you correctly<br>I just need to insert that if-line between every consequetive computation step?<br>
<br># Loop over jobs<br>for i in range(jobs):<br> # Insert job into dictinoary to keep track of it<br> proc[i] = grass.start_command('r.slope.aspect',<br> elevation='elev_' + str(i),<br>
slope='slope_' + str(i))<br><br> # Probably here I have to wait until slope_i is created??<br> if i % workers is 0:<br> for j in range(workers):<br> proc.[i - j].wait()<br>
# How would you do that? In my case these are a dozend<br> # consequetive grass commands (output1=input2...) <br> # and some thousand "jobs" which I want to loop over<br><br> proc[i] = grass.start_command('r.cost',<br>
input='slope_' + str(i),<br> output='costraster_' + str(i),<br> coordinate = "123,123")<br> if i % workers is 0:<br>
for j in range(workers):<br> proc.[i - j].wait()<br><br># Make sure all workers are finished.<br>for i in range(jobs):<br> if proc[i].wait() is not 0:<br> grass.fatal(_('Problem running analysis on evel_' + str(i) + '.')<span><font color="#888888"><br>
<br><br>/johannes</font></span><div><div><br><br><br><div class="gmail_quote">On Tue, Aug 7, 2012 at 10:21 AM, Daniel Lee <span dir="ltr"><<a href="mailto:lee@isi-solutions.org" target="_blank">lee@isi-solutions.org</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Hi Johannes,<br><br>Certainly it's possible, it'd just be a question of how you make your loop. Inside the loop of your jobs you could evaluate whether it's an odd or even job and check if the previous job has been finished, then start the next one. I'm not sure what your code looks like, but I've condensed Hamish's script for my own reference into an example gist so that I can remember some of the cool tricks. I don't know if it would help you to look at it since you're already seen the original script, but if you want to take a look feel free to:<br>
<br><a href="https://gist.github.com/3282580" target="_blank">https://gist.github.com/3282580</a><br><br>I think you'll basically have to work with the modulo operator a bit. If you use that, you can possibly reduce a lot of nested for loops into a for loop with an if-elif-else query.<div>
<br>
<br>Best,<br>Daniel<br clear="all">
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px"><span style="font-size:13px;font-family:arial,sans-serif"></span></p><p>--<br></p><p>B.Sc. Daniel Lee<br>
Geschäftsführung für Forschung und Entwicklung<br>ISIS - International Solar Information Solutions GbR<br>Vertreten durch: Daniel Lee, Nepomuk Reinhard und Nils Räder<br></p><p>Softwarecenter 3<br>35037 Marburg<br>Festnetz: <a value="+4964213796256" style="color:rgb(28,81,168)">+49 6421 379 6256</a><br>
Mobil: <a value="+4917661277269" style="color:rgb(28,81,168)">+49 176 6127 7269</a><br>E-Mail: <a href="mailto:Lee@isi-solutions.org" style="color:rgb(28,81,168)" target="_blank">Lee@isi-solutions.org</a><br>Web: <a href="http://www.isi-solutions.org/" style="color:rgb(28,81,168)" target="_blank">http://www.isi-solutions.org</a></p>
<p></p><br>
<br><br></div><div><div><div class="gmail_quote">2012/8/7 Johannes Radinger <span dir="ltr"><<a href="mailto:johannesradinger@gmail.com" target="_blank">johannesradinger@gmail.com</a>></span><br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Hi,<br><br><div class="gmail_quote"><div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
for an example of grass.start_command() for parallelizing a bunch<br>
of r.cost runs, see v.surf.icw(.py) in grass7 addons:<br>
<br>
<a href="https://trac.osgeo.org/grass/browser/grass-addons/grass7/vector/v.surf.icw/v.surf.icw.py" target="_blank">https://trac.osgeo.org/grass/browser/grass-addons/grass7/vector/v.surf.icw/v.surf.icw.py</a><br>
<br></blockquote></div>thank you for that example. I think it explains it very well how it works to assign<br>multiple r.cost runs to single processes with grass.start_command.<br>I am just wondering how it is done when there are multiple consecutive processes<br>
in the for loop. In your example (<a href="http://v.surf.icw.py" target="_blank">v.surf.icw.py</a>) for each step (e.g. r.cost (line 271), r.mapcalc (298))<br>an separate for loop is started...Is there a way to combine the steps etc. in a function (e.g. combination<br>
of r.cost and mapcalc) and launch that function in a way like grass.start_command in a single loop?<br>If possible that would probably save code lines and might be a little more clear (at least to me). <br>I am just asking because one of my skripts which is still in "serial mode" involves lots of steps inside the for loop.<br>
This would create in parallel at least a dozen for loops which might appear very unclear.<br><br>Anyway I think the parallelization can really save computation time of my skripts... :)<span><font color="#888888"><br>
<br>/Johannes<br></font></span></div>
</blockquote></div><br>
</div></div></blockquote></div><br>
</div></div></blockquote></div><br>
</div></div></blockquote></div><br>