<div dir="ltr"><div><div><div><div><div><div><div>Hi Lorenzo:<br><br></div>This is more of a question for the python community. However, a couple things I have noticed. Pandas tends to be much slower than working in numpy directly.<br></div>I never saw an improvement in timings when using Pool(). What I do is utilize Process() and Queue() or JoinableQueue() from the multiprocessing library.<br><br></div>You can setup a pool of workers that all read from the same input queue. Then you can just feed your data into the queue so the workers can process the maps.<br></div><br></div></div></div><div><div><div><div><div><br><div><br><br></div></div></div></div></div></div></div><div class="gmail_extra"><br clear="all"><div><div class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><span><pre>Jerl Simpson
Sr. Systems Engineer
Weather Trends International
<a href="http://www.weathertrends360.com/" target="_blank">http://www.weathertrends360.com/</a>
This communication is privileged and may contain confidential information.<br>It's intended only for the use of the person or entity named above.<br>If you are not the intended recipient, do not distribute or copy this communication.<br>If you have received this communication in error, <br>please notify the sender immediately and return the original to the email address above.<br>© Copyright 2016 Weather Trends International, Inc.</pre></span></div></div></div></div></div>
<br><div class="gmail_quote">On Wed, Mar 2, 2016 at 5:44 PM, Lorenzo Bottaccioli <span dir="ltr"><<a href="mailto:lorenzo.bottaccioli@gmail.com" target="_blank">lorenzo.bottaccioli@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><p style="margin:0px 0px 1em;padding:0px;border:0px;font-size:15px;clear:both;color:rgb(34,36,38);font-family:Arial,'Helvetica Neue',Helvetica,sans-serif;line-height:19.5px">Hi,<br>I'm trying to parallelize a code for raster calculation with Gdal_calc.py, but i have relay bad results. I need to perform several raster operation like FILE_out=FILA_a<em style="margin:0px;padding:0px;border:0px">k1+FILE_b</em>k2.</p><p style="margin:0px 0px 1em;padding:0px;border:0px;font-size:15px;clear:both;color:rgb(34,36,38);font-family:Arial,'Helvetica Neue',Helvetica,sans-serif;line-height:19.5px">This is the code I'm usign:</p><pre style="margin-top:0px;margin-bottom:1em;padding:5px;border:0px;font-size:13px;overflow:auto;width:auto;max-height:600px;font-family:Consolas,Menlo,Monaco,'Lucida Console','Liberation Mono','DejaVu Sans Mono','Bitstream Vera Sans Mono','Courier New',monospace,sans-serif;color:rgb(57,51,24);word-wrap:normal;background-color:rgb(238,238,238)"><code style="margin:0px;padding:0px;border:0px;font-family:Consolas,Menlo,Monaco,'Lucida Console','Liberation Mono','DejaVu Sans Mono','Bitstream Vera Sans Mono','Courier New',monospace,sans-serif;white-space:inherit"><span style="margin:0px;padding:0px;border:0px;color:rgb(0,0,139)">import</span><span style="margin:0px;padding:0px;border:0px;color:black"> pandas </span><span style="margin:0px;padding:0px;border:0px;color:rgb(0,0,139)">as</span><span style="margin:0px;padding:0px;border:0px;color:black"> pd
</span><span style="margin:0px;padding:0px;border:0px;color:rgb(0,0,139)">import</span><span style="margin:0px;padding:0px;border:0px;color:black"> os
</span><span style="margin:0px;padding:0px;border:0px;color:rgb(0,0,139)">import</span><span style="margin:0px;padding:0px;border:0px;color:black"> time
</span><span style="margin:0px;padding:0px;border:0px;color:rgb(0,0,139)">from</span><span style="margin:0px;padding:0px;border:0px;color:black"> multiprocessing </span><span style="margin:0px;padding:0px;border:0px;color:rgb(0,0,139)">import</span><span style="margin:0px;padding:0px;border:0px;color:black"> </span><span style="margin:0px;padding:0px;border:0px;color:rgb(43,145,175)">Pool</span><span style="margin:0px;padding:0px;border:0px;color:black">
df </span><span style="margin:0px;padding:0px;border:0px;color:black">=</span><span style="margin:0px;padding:0px;border:0px;color:black"> pd</span><span style="margin:0px;padding:0px;border:0px;color:black">.</span><span style="margin:0px;padding:0px;border:0px;color:black">read_csv</span><span style="margin:0px;padding:0px;border:0px;color:black">(</span><span style="margin:0px;padding:0px;border:0px;color:maroon">'input.csv'</span><span style="margin:0px;padding:0px;border:0px;color:black">,</span><span style="margin:0px;padding:0px;border:0px;color:black"> sep</span><span style="margin:0px;padding:0px;border:0px;color:black">=</span><span style="margin:0px;padding:0px;border:0px;color:maroon">";"</span><span style="margin:0px;padding:0px;border:0px;color:black">,</span><span style="margin:0px;padding:0px;border:0px;color:black"> index_col</span><span style="margin:0px;padding:0px;border:0px;color:black">=</span><span style="margin:0px;padding:0px;border:0px;color:maroon">'Date Time'</span><span style="margin:0px;padding:0px;border:0px;color:black">,</span><span style="margin:0px;padding:0px;border:0px;color:black"> decimal</span><span style="margin:0px;padding:0px;border:0px;color:black">=</span><span style="margin:0px;padding:0px;border:0px;color:maroon">','</span><span style="margin:0px;padding:0px;border:0px;color:black">)</span><span style="margin:0px;padding:0px;border:0px;color:black">
df</span><span style="margin:0px;padding:0px;border:0px;color:black">.</span><span style="margin:0px;padding:0px;border:0px;color:black">index </span><span style="margin:0px;padding:0px;border:0px;color:black">=</span><span style="margin:0px;padding:0px;border:0px;color:black"> pd</span><span style="margin:0px;padding:0px;border:0px;color:black">.</span><span style="margin:0px;padding:0px;border:0px;color:black">to_datetime</span><span style="margin:0px;padding:0px;border:0px;color:black">(</span><span style="margin:0px;padding:0px;border:0px;color:black">df</span><span style="margin:0px;padding:0px;border:0px;color:black">.</span><span style="margin:0px;padding:0px;border:0px;color:black">index</span><span style="margin:0px;padding:0px;border:0px;color:black">,</span><span style="margin:0px;padding:0px;border:0px;color:black"> unit</span><span style="margin:0px;padding:0px;border:0px;color:black">=</span><span style="margin:0px;padding:0px;border:0px;color:maroon">'s'</span><span style="margin:0px;padding:0px;border:0px;color:black">)</span><span style="margin:0px;padding:0px;border:0px;color:black">
start_time </span><span style="margin:0px;padding:0px;border:0px;color:black">=</span><span style="margin:0px;padding:0px;border:0px;color:black"> time</span><span style="margin:0px;padding:0px;border:0px;color:black">.</span><span style="margin:0px;padding:0px;border:0px;color:black">time</span><span style="margin:0px;padding:0px;border:0px;color:black">()</span><span style="margin:0px;padding:0px;border:0px;color:black">
pool</span><span style="margin:0px;padding:0px;border:0px;color:black">=</span><span style="margin:0px;padding:0px;border:0px;color:rgb(43,145,175)">Pool</span><span style="margin:0px;padding:0px;border:0px;color:black">(</span><span style="margin:0px;padding:0px;border:0px;color:black">processes</span><span style="margin:0px;padding:0px;border:0px;color:black">=</span><span style="margin:0px;padding:0px;border:0px;color:maroon">8</span><span style="margin:0px;padding:0px;border:0px;color:black">)</span><span style="margin:0px;padding:0px;border:0px;color:black">
pool</span><span style="margin:0px;padding:0px;border:0px;color:black">.</span><span style="margin:0px;padding:0px;border:0px;color:black">map</span><span style="margin:0px;padding:0px;border:0px;color:black">(</span><span style="margin:0px;padding:0px;border:0px;color:black">mapcalc</span><span style="margin:0px;padding:0px;border:0px;color:black">,[</span><span style="margin:0px;padding:0px;border:0px;color:black">df</span><span style="margin:0px;padding:0px;border:0px;color:black">.</span><span style="margin:0px;padding:0px;border:0px;color:black">iloc</span><span style="margin:0px;padding:0px;border:0px;color:black">[</span><span style="margin:0px;padding:0px;border:0px;color:black">i</span><span style="margin:0px;padding:0px;border:0px;color:black">*</span><span style="margin:0px;padding:0px;border:0px;color:maroon">20</span><span style="margin:0px;padding:0px;border:0px;color:black">:(</span><span style="margin:0px;padding:0px;border:0px;color:black">i</span><span style="margin:0px;padding:0px;border:0px;color:black">+</span><span style="margin:0px;padding:0px;border:0px;color:maroon">1</span><span style="margin:0px;padding:0px;border:0px;color:black">)*</span><span style="margin:0px;padding:0px;border:0px;color:maroon">20</span><span style="margin:0px;padding:0px;border:0px;color:black">]</span><span style="margin:0px;padding:0px;border:0px;color:black"> </span><span style="margin:0px;padding:0px;border:0px;color:rgb(0,0,139)">for</span><span style="margin:0px;padding:0px;border:0px;color:black"> i </span><span style="margin:0px;padding:0px;border:0px;color:rgb(0,0,139)">in</span><span style="margin:0px;padding:0px;border:0px;color:black"> range</span><span style="margin:0px;padding:0px;border:0px;color:black">(</span><span style="margin:0px;padding:0px;border:0px;color:black">len</span><span style="margin:0px;padding:0px;border:0px;color:black">(</span><span style="margin:0px;padding:0px;border:0px;color:black">df</span><span style="margin:0px;padding:0px;border:0px;color:black">.</span><span style="margin:0px;padding:0px;border:0px;color:black">index</span><span style="margin:0px;padding:0px;border:0px;color:black">)/</span><span style="margin:0px;padding:0px;border:0px;color:maroon">20</span><span style="margin:0px;padding:0px;border:0px;color:black">+</span><span style="margin:0px;padding:0px;border:0px;color:maroon">1</span><span style="margin:0px;padding:0px;border:0px;color:black">)])</span><span style="margin:0px;padding:0px;border:0px;color:black">
pool</span><span style="margin:0px;padding:0px;border:0px;color:black">.</span><span style="margin:0px;padding:0px;border:0px;color:black">close</span><span style="margin:0px;padding:0px;border:0px;color:black">()</span><span style="margin:0px;padding:0px;border:0px;color:black">
pool</span><span style="margin:0px;padding:0px;border:0px;color:black">.</span><span style="margin:0px;padding:0px;border:0px;color:black">join</span><span style="margin:0px;padding:0px;border:0px;color:black">()</span><span style="margin:0px;padding:0px;border:0px;color:black">
</span><span style="margin:0px;padding:0px;border:0px;color:rgb(0,0,139)">print</span><span style="margin:0px;padding:0px;border:0px;color:black">(</span><span style="margin:0px;padding:0px;border:0px;color:maroon">"--- %s seconds ---"</span><span style="margin:0px;padding:0px;border:0px;color:black"> </span><span style="margin:0px;padding:0px;border:0px;color:black">%</span><span style="margin:0px;padding:0px;border:0px;color:black"> </span><span style="margin:0px;padding:0px;border:0px;color:black">(</span><span style="margin:0px;padding:0px;border:0px;color:black">time</span><span style="margin:0px;padding:0px;border:0px;color:black">.</span><span style="margin:0px;padding:0px;border:0px;color:black">time</span><span style="margin:0px;padding:0px;border:0px;color:black">()</span><span style="margin:0px;padding:0px;border:0px;color:black"> </span><span style="margin:0px;padding:0px;border:0px;color:black">-</span><span style="margin:0px;padding:0px;border:0px;color:black"> start_time</span><span style="margin:0px;padding:0px;border:0px;color:black">))</span><span style="margin:0px;padding:0px;border:0px;color:black">
</span><span style="margin:0px;padding:0px;border:0px;color:rgb(0,0,139)">def</span><span style="margin:0px;padding:0px;border:0px;color:black"> mapcalc</span><span style="margin:0px;padding:0px;border:0px;color:black">(</span><span style="margin:0px;padding:0px;border:0px;color:black">df</span><span style="margin:0px;padding:0px;border:0px;color:black">):</span><span style="margin:0px;padding:0px;border:0px;color:black">
month</span><span style="margin:0px;padding:0px;border:0px;color:black">={</span><span style="margin:0px;padding:0px;border:0px;color:maroon">1</span><span style="margin:0px;padding:0px;border:0px;color:black">:</span><span style="margin:0px;padding:0px;border:0px;color:maroon">'17'</span><span style="margin:0px;padding:0px;border:0px;color:black">,</span><span style="margin:0px;padding:0px;border:0px;color:maroon">2</span><span style="margin:0px;padding:0px;border:0px;color:black">:</span><span style="margin:0px;padding:0px;border:0px;color:maroon">'47'</span><span style="margin:0px;padding:0px;border:0px;color:black">,</span><span style="margin:0px;padding:0px;border:0px;color:maroon">3</span><span style="margin:0px;padding:0px;border:0px;color:black">:</span><span style="margin:0px;padding:0px;border:0px;color:maroon">'75'</span><span style="margin:0px;padding:0px;border:0px;color:black">,</span><span style="margin:0px;padding:0px;border:0px;color:maroon">4</span><span style="margin:0px;padding:0px;border:0px;color:black">:</span><span style="margin:0px;padding:0px;border:0px;color:maroon">'105'</span><span style="margin:0px;padding:0px;border:0px;color:black">,</span><span style="margin:0px;padding:0px;border:0px;color:maroon">5</span><span style="margin:0px;padding:0px;border:0px;color:black">:</span><span style="margin:0px;padding:0px;border:0px;color:maroon">'135'</span><span style="margin:0px;padding:0px;border:0px;color:black">,</span><span style="margin:0px;padding:0px;border:0px;color:maroon">6</span><span style="margin:0px;padding:0px;border:0px;color:black">:</span><span style="margin:0px;padding:0px;border:0px;color:maroon">'162'</span><span style="margin:0px;padding:0px;border:0px;color:black">,</span><span style="margin:0px;padding:0px;border:0px;color:maroon">7</span><span style="margin:0px;padding:0px;border:0px;color:black">:</span><span style="margin:0px;padding:0px;border:0px;color:maroon">'198'</span><span style="margin:0px;padding:0px;border:0px;color:black">,</span><span style="margin:0px;padding:0px;border:0px;color:maroon">8</span><span style="margin:0px;padding:0px;border:0px;color:black">:</span><span style="margin:0px;padding:0px;border:0px;color:maroon">'228'</span><span style="margin:0px;padding:0px;border:0px;color:black">,</span><span style="margin:0px;padding:0px;border:0px;color:maroon">9</span><span style="margin:0px;padding:0px;border:0px;color:black">:</span><span style="margin:0px;padding:0px;border:0px;color:maroon">'258'</span><span style="margin:0px;padding:0px;border:0px;color:black">,</span><span style="margin:0px;padding:0px;border:0px;color:maroon">10</span><span style="margin:0px;padding:0px;border:0px;color:black">:</span><span style="margin:0px;padding:0px;border:0px;color:maroon">'288'</span><span style="margin:0px;padding:0px;border:0px;color:black">,</span><span style="margin:0px;padding:0px;border:0px;color:maroon">11</span><span style="margin:0px;padding:0px;border:0px;color:black">:</span><span style="margin:0px;padding:0px;border:0px;color:maroon">'318'</span><span style="margin:0px;padding:0px;border:0px;color:black">,</span><span style="margin:0px;padding:0px;border:0px;color:maroon">12</span><span style="margin:0px;padding:0px;border:0px;color:black">:</span><span style="margin:0px;padding:0px;border:0px;color:maroon">'344'</span><span style="margin:0px;padding:0px;border:0px;color:black">}</span><span style="margin:0px;padding:0px;border:0px;color:black">
hour</span><span style="margin:0px;padding:0px;border:0px;color:black">={</span><span style="margin:0px;padding:0px;border:0px;color:maroon">4</span><span style="margin:0px;padding:0px;border:0px;color:black">:</span><span style="margin:0px;padding:0px;border:0px;color:maroon">'04'</span><span style="margin:0px;padding:0px;border:0px;color:black">,</span><span style="margin:0px;padding:0px;border:0px;color:maroon">5</span><span style="margin:0px;padding:0px;border:0px;color:black">:</span><span style="margin:0px;padding:0px;border:0px;color:maroon">'05'</span><span style="margin:0px;padding:0px;border:0px;color:black">,</span><span style="margin:0px;padding:0px;border:0px;color:maroon">6</span><span style="margin:0px;padding:0px;border:0px;color:black">:</span><span style="margin:0px;padding:0px;border:0px;color:maroon">'06'</span><span style="margin:0px;padding:0px;border:0px;color:black">,</span><span style="margin:0px;padding:0px;border:0px;color:maroon">7</span><span style="margin:0px;padding:0px;border:0px;color:black">:</span><span style="margin:0px;padding:0px;border:0px;color:maroon">'07'</span><span style="margin:0px;padding:0px;border:0px;color:black">,</span><span style="margin:0px;padding:0px;border:0px;color:maroon">8</span><span style="margin:0px;padding:0px;border:0px;color:black">:</span><span style="margin:0px;padding:0px;border:0px;color:maroon">'08'</span><span style="margin:0px;padding:0px;border:0px;color:black">,</span><span style="margin:0px;padding:0px;border:0px;color:maroon">9</span><span style="margin:0px;padding:0px;border:0px;color:black">:</span><span style="margin:0px;padding:0px;border:0px;color:maroon">'09'</span><span style="margin:0px;padding:0px;border:0px;color:black">,</span><span style="margin:0px;padding:0px;border:0px;color:maroon">10</span><span style="margin:0px;padding:0px;border:0px;color:black">:</span><span style="margin:0px;padding:0px;border:0px;color:maroon">'10'</span><span style="margin:0px;padding:0px;border:0px;color:black">,</span><span style="margin:0px;padding:0px;border:0px;color:maroon">11</span><span style="margin:0px;padding:0px;border:0px;color:black">:</span><span style="margin:0px;padding:0px;border:0px;color:maroon">'11'</span><span style="margin:0px;padding:0px;border:0px;color:black">,</span><span style="margin:0px;padding:0px;border:0px;color:maroon">12</span><span style="margin:0px;padding:0px;border:0px;color:black">:</span><span style="margin:0px;padding:0px;border:0px;color:maroon">'12'</span><span style="margin:0px;padding:0px;border:0px;color:black">,</span><span style="margin:0px;padding:0px;border:0px;color:maroon">13</span><span style="margin:0px;padding:0px;border:0px;color:black">:</span><span style="margin:0px;padding:0px;border:0px;color:maroon">'13'</span><span style="margin:0px;padding:0px;border:0px;color:black">,</span><span style="margin:0px;padding:0px;border:0px;color:maroon">14</span><span style="margin:0px;padding:0px;border:0px;color:black">:</span><span style="margin:0px;padding:0px;border:0px;color:maroon">'14'</span><span style="margin:0px;padding:0px;border:0px;color:black">,</span><span style="margin:0px;padding:0px;border:0px;color:maroon">15</span><span style="margin:0px;padding:0px;border:0px;color:black">:</span><span style="margin:0px;padding:0px;border:0px;color:maroon">'15'</span><span style="margin:0px;padding:0px;border:0px;color:black">,</span><span style="margin:0px;padding:0px;border:0px;color:maroon">16</span><span style="margin:0px;padding:0px;border:0px;color:black">:</span><span style="margin:0px;padding:0px;border:0px;color:maroon">'16'</span><span style="margin:0px;padding:0px;border:0px;color:black">,</span><span style="margin:0px;padding:0px;border:0px;color:maroon">17</span><span style="margin:0px;padding:0px;border:0px;color:black">:</span><span style="margin:0px;padding:0px;border:0px;color:maroon">'17'</span><span style="margin:0px;padding:0px;border:0px;color:black">,</span><span style="margin:0px;padding:0px;border:0px;color:maroon">18</span><span style="margin:0px;padding:0px;border:0px;color:black">:</span><span style="margin:0px;padding:0px;border:0px;color:maroon">'18'</span><span style="margin:0px;padding:0px;border:0px;color:black">,</span><span style="margin:0px;padding:0px;border:0px;color:maroon">19</span><span style="margin:0px;padding:0px;border:0px;color:black">:</span><span style="margin:0px;padding:0px;border:0px;color:maroon">'19'</span><span style="margin:0px;padding:0px;border:0px;color:black">,</span><span style="margin:0px;padding:0px;border:0px;color:maroon">20</span><span style="margin:0px;padding:0px;border:0px;color:black">:</span><span style="margin:0px;padding:0px;border:0px;color:maroon">'20'</span><span style="margin:0px;padding:0px;border:0px;color:black">,</span><span style="margin:0px;padding:0px;border:0px;color:maroon">21</span><span style="margin:0px;padding:0px;border:0px;color:black">:</span><span style="margin:0px;padding:0px;border:0px;color:maroon">'21'</span><span style="margin:0px;padding:0px;border:0px;color:black">,</span><span style="margin:0px;padding:0px;border:0px;color:maroon">22</span><span style="margin:0px;padding:0px;border:0px;color:black">:</span><span style="margin:0px;padding:0px;border:0px;color:maroon">'22'</span><span style="margin:0px;padding:0px;border:0px;color:black">}</span><span style="margin:0px;padding:0px;border:0px;color:black">
minute</span><span style="margin:0px;padding:0px;border:0px;color:black">={</span><span style="margin:0px;padding:0px;border:0px;color:maroon">0</span><span style="margin:0px;padding:0px;border:0px;color:black">:</span><span style="margin:0px;padding:0px;border:0px;color:maroon">'00'</span><span style="margin:0px;padding:0px;border:0px;color:black">,</span><span style="margin:0px;padding:0px;border:0px;color:maroon">15</span><span style="margin:0px;padding:0px;border:0px;color:black">:</span><span style="margin:0px;padding:0px;border:0px;color:maroon">'15'</span><span style="margin:0px;padding:0px;border:0px;color:black">,</span><span style="margin:0px;padding:0px;border:0px;color:maroon">30</span><span style="margin:0px;padding:0px;border:0px;color:black">:</span><span style="margin:0px;padding:0px;border:0px;color:maroon">'30'</span><span style="margin:0px;padding:0px;border:0px;color:black">,</span><span style="margin:0px;padding:0px;border:0px;color:maroon">45</span><span style="margin:0px;padding:0px;border:0px;color:black">:</span><span style="margin:0px;padding:0px;border:0px;color:maroon">'45'</span><span style="margin:0px;padding:0px;border:0px;color:black">}</span><span style="margin:0px;padding:0px;border:0px;color:black">
directory</span><span style="margin:0px;padding:0px;border:0px;color:black">=</span><span style="margin:0px;padding:0px;border:0px;color:maroon">'/home/user/Raster/'</span><span style="margin:0px;padding:0px;border:0px;color:black">
tmp</span><span style="margin:0px;padding:0px;border:0px;color:black">=</span><span style="margin:0px;padding:0px;border:0px;color:maroon">'/home/usr/tmp/'</span><span style="margin:0px;padding:0px;border:0px;color:black">
</span><span style="margin:0px;padding:0px;border:0px;color:rgb(0,0,139)">for</span><span style="margin:0px;padding:0px;border:0px;color:black"> i </span><span style="margin:0px;padding:0px;border:0px;color:rgb(0,0,139)">in</span><span style="margin:0px;padding:0px;border:0px;color:black"> df</span><span style="margin:0px;padding:0px;border:0px;color:black">.</span><span style="margin:0px;padding:0px;border:0px;color:black">index</span><span style="margin:0px;padding:0px;border:0px;color:black">:</span><span style="margin:0px;padding:0px;border:0px;color:black">
</span><span style="margin:0px;padding:0px;border:0px;color:rgb(0,0,139)">if</span><span style="margin:0px;padding:0px;border:0px;color:black"> </span><span style="margin:0px;padding:0px;border:0px;color:maroon">4</span><span style="margin:0px;padding:0px;border:0px;color:black"><=</span><span style="margin:0px;padding:0px;border:0px;color:black">i</span><span style="margin:0px;padding:0px;border:0px;color:black">.</span><span style="margin:0px;padding:0px;border:0px;color:black">hour</span><span style="margin:0px;padding:0px;border:0px;color:black"><</span><span style="margin:0px;padding:0px;border:0px;color:maroon">22</span><span style="margin:0px;padding:0px;border:0px;color:black">:</span><span style="margin:0px;padding:0px;border:0px;color:black">
</span><span style="margin:0px;padding:0px;border:0px;color:gray">#try:</span><span style="margin:0px;padding:0px;border:0px;color:black">
timeg</span><span style="margin:0px;padding:0px;border:0px;color:black">=</span><span style="margin:0px;padding:0px;border:0px;color:black">time</span><span style="margin:0px;padding:0px;border:0px;color:black">.</span><span style="margin:0px;padding:0px;border:0px;color:black">time</span><span style="margin:0px;padding:0px;border:0px;color:black">()</span><span style="margin:0px;padding:0px;border:0px;color:black">
os</span><span style="margin:0px;padding:0px;border:0px;color:black">.</span><span style="margin:0px;padding:0px;border:0px;color:black">system</span><span style="margin:0px;padding:0px;border:0px;color:black">(</span><span style="margin:0px;padding:0px;border:0px;color:maroon">'gdal_calc.py -A '</span><span style="margin:0px;padding:0px;border:0px;color:black">+</span><span style="margin:0px;padding:0px;border:0px;color:black">directory</span><span style="margin:0px;padding:0px;border:0px;color:black">+</span><span style="margin:0px;padding:0px;border:0px;color:maroon">'filea_'</span><span style="margin:0px;padding:0px;border:0px;color:black">+</span><span style="margin:0px;padding:0px;border:0px;color:black">month</span><span style="margin:0px;padding:0px;border:0px;color:black">[</span><span style="margin:0px;padding:0px;border:0px;color:black">i</span><span style="margin:0px;padding:0px;border:0px;color:black">.</span><span style="margin:0px;padding:0px;border:0px;color:black">month</span><span style="margin:0px;padding:0px;border:0px;color:black">]+</span><span style="margin:0px;padding:0px;border:0px;color:maroon">'_'</span><span style="margin:0px;padding:0px;border:0px;color:black">+</span><span style="margin:0px;padding:0px;border:0px;color:black">hour</span><span style="margin:0px;padding:0px;border:0px;color:black">[</span><span style="margin:0px;padding:0px;border:0px;color:black">i</span><span style="margin:0px;padding:0px;border:0px;color:black">.</span><span style="margin:0px;padding:0px;border:0px;color:black">hour</span><span style="margin:0px;padding:0px;border:0px;color:black">]+</span><span style="margin:0px;padding:0px;border:0px;color:black">minute</span><span style="margin:0px;padding:0px;border:0px;color:black">[</span><span style="margin:0px;padding:0px;border:0px;color:black">i</span><span style="margin:0px;padding:0px;border:0px;color:black">.</span><span style="margin:0px;padding:0px;border:0px;color:black">minute</span><span style="margin:0px;padding:0px;border:0px;color:black">]+</span><span style="margin:0px;padding:0px;border:0px;color:maroon">' -B '</span><span style="margin:0px;padding:0px;border:0px;color:black">+</span><span style="margin:0px;padding:0px;border:0px;color:black">directory</span><span style="margin:0px;padding:0px;border:0px;color:black">+</span><span style="margin:0px;padding:0px;border:0px;color:maroon">'fileb_'</span><span style="margin:0px;padding:0px;border:0px;color:black">+</span><span style="margin:0px;padding:0px;border:0px;color:black">month</span><span style="margin:0px;padding:0px;border:0px;color:black">[</span><span style="margin:0px;padding:0px;border:0px;color:black">i</span><span style="margin:0px;padding:0px;border:0px;color:black">.</span><span style="margin:0px;padding:0px;border:0px;color:black">month</span><span style="margin:0px;padding:0px;border:0px;color:black">]+</span><span style="margin:0px;padding:0px;border:0px;color:maroon">'_'</span><span style="margin:0px;padding:0px;border:0px;color:black">+</span><span style="margin:0px;padding:0px;border:0px;color:black">hour</span><span style="margin:0px;padding:0px;border:0px;color:black">[</span><span style="margin:0px;padding:0px;border:0px;color:black">i</span><span style="margin:0px;padding:0px;border:0px;color:black">.</span><span style="margin:0px;padding:0px;border:0px;color:black">hour</span><span style="margin:0px;padding:0px;border:0px;color:black">]+</span><span style="margin:0px;padding:0px;border:0px;color:black">minute</span><span style="margin:0px;padding:0px;border:0px;color:black">[</span><span style="margin:0px;padding:0px;border:0px;color:black">i</span><span style="margin:0px;padding:0px;border:0px;color:black">.</span><span style="margin:0px;padding:0px;border:0px;color:black">minute</span><span style="margin:0px;padding:0px;border:0px;color:black">]+</span><span style="margin:0px;padding:0px;border:0px;color:maroon">' --outfile='</span><span style="margin:0px;padding:0px;border:0px;color:black">+</span><span style="margin:0px;padding:0px;border:0px;color:black">tmp</span><span style="margin:0px;padding:0px;border:0px;color:black">+</span><span style="margin:0px;padding:0px;border:0px;color:black">str</span><span style="margin:0px;padding:0px;border:0px;color:black">(</span><span style="margin:0px;padding:0px;border:0px;color:black">i</span><span style="margin:0px;padding:0px;border:0px;color:black">.</span><span style="margin:0px;padding:0px;border:0px;color:black">date</span><span style="margin:0px;padding:0px;border:0px;color:black">())+</span><span style="margin:0px;padding:0px;border:0px;color:maroon">'_'</span><span style="margin:0px;padding:0px;border:0px;color:black">+</span><span style="margin:0px;padding:0px;border:0px;color:black">str</span><span style="margin:0px;padding:0px;border:0px;color:black">(</span><span style="margin:0px;padding:0px;border:0px;color:black">i</span><span style="margin:0px;padding:0px;border:0px;color:black">.</span><span style="margin:0px;padding:0px;border:0px;color:black">time</span><span style="margin:0px;padding:0px;border:0px;color:black">())+</span><span style="margin:0px;padding:0px;border:0px;color:maroon">' --calc=A*'</span><span style="margin:0px;padding:0px;border:0px;color:black">+</span><span style="margin:0px;padding:0px;border:0px;color:black">str</span><span style="margin:0px;padding:0px;border:0px;color:black">(</span><span style="margin:0px;padding:0px;border:0px;color:black">df</span><span style="margin:0px;padding:0px;border:0px;color:black">.</span><span style="margin:0px;padding:0px;border:0px;color:black">ix</span><span style="margin:0px;padding:0px;border:0px;color:black">[</span><span style="margin:0px;padding:0px;border:0px;color:black">i</span><span style="margin:0px;padding:0px;border:0px;color:black">,</span><span style="margin:0px;padding:0px;border:0px;color:maroon">'k1'</span><span style="margin:0px;padding:0px;border:0px;color:black">])+</span><span style="margin:0px;padding:0px;border:0px;color:maroon">'+B*'</span><span style="margin:0px;padding:0px;border:0px;color:black">+</span><span style="margin:0px;padding:0px;border:0px;color:black">str</span><span style="margin:0px;padding:0px;border:0px;color:black">(</span><span style="margin:0px;padding:0px;border:0px;color:black">df</span><span style="margin:0px;padding:0px;border:0px;color:black">.</span><span style="margin:0px;padding:0px;border:0px;color:black">ix</span><span style="margin:0px;padding:0px;border:0px;color:black">[</span><span style="margin:0px;padding:0px;border:0px;color:black">i</span><span style="margin:0px;padding:0px;border:0px;color:black">,</span><span style="margin:0px;padding:0px;border:0px;color:maroon">'k2'</span><span style="margin:0px;padding:0px;border:0px;color:black">]))</span><span style="margin:0px;padding:0px;border:0px;color:black">
</span><span style="margin:0px;padding:0px;border:0px;color:rgb(0,0,139)">print</span><span style="margin:0px;padding:0px;border:0px;color:black">(</span><span style="margin:0px;padding:0px;border:0px;color:black">i</span><span style="margin:0px;padding:0px;border:0px;color:black">,</span><span style="margin:0px;padding:0px;border:0px;color:maroon">"--- %s seconds ---"</span><span style="margin:0px;padding:0px;border:0px;color:black"> </span><span style="margin:0px;padding:0px;border:0px;color:black">%</span><span style="margin:0px;padding:0px;border:0px;color:black"> </span><span style="margin:0px;padding:0px;border:0px;color:black">(</span><span style="margin:0px;padding:0px;border:0px;color:black">time</span><span style="margin:0px;padding:0px;border:0px;color:black">.</span><span style="margin:0px;padding:0px;border:0px;color:black">time</span><span style="margin:0px;padding:0px;border:0px;color:black">()</span><span style="margin:0px;padding:0px;border:0px;color:black"> </span><span style="margin:0px;padding:0px;border:0px;color:black">-</span><span style="margin:0px;padding:0px;border:0px;color:black"> timeg</span><span style="margin:0px;padding:0px;border:0px;color:black">))</span></code></pre><p style="margin:0px 0px 1em;padding:0px;border:0px;font-size:15px;clear:both;color:rgb(34,36,38);font-family:Arial,'Helvetica Neue',Helvetica,sans-serif;line-height:19.5px">If i run the code with out parallelization it takes around 650s to complete the calculation. Each process of the for loop is executed in ~10s. If i run with parallelization it takes ~900s to complete the procces and each process of the for loop it takes ~30s.</p><p style="margin:0px 0px 1em;padding:0px;border:0px;font-size:15px;clear:both;color:rgb(34,36,38);font-family:Arial,'Helvetica Neue',Helvetica,sans-serif;line-height:19.5px">How is that? how can i Fix this?</p><p style="margin:0px 0px 1em;padding:0px;border:0px;font-size:15px;clear:both;color:rgb(34,36,38);font-family:Arial,'Helvetica Neue',Helvetica,sans-serif;line-height:19.5px">Best L</p></div>
<br>_______________________________________________<br>
gdal-dev mailing list<br>
<a href="mailto:gdal-dev@lists.osgeo.org">gdal-dev@lists.osgeo.org</a><br>
<a href="http://lists.osgeo.org/mailman/listinfo/gdal-dev" rel="noreferrer" target="_blank">http://lists.osgeo.org/mailman/listinfo/gdal-dev</a><br></blockquote></div><br></div>