<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Menlo;
panose-1:0 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:Georgia;
panose-1:2 4 5 2 5 4 5 2 3 3;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:70.85pt 70.85pt 70.85pt 70.85pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="NO-BOK" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">Hi Mark,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">Don`t know if that is of any help, but:<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">Have you tried the igraph package for very customized / sophisticated network analysis (http://igraph.org/redirect.html)?<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">It plays nicely with R, python, and C and therefor also with GRASS.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">What I did (now in several cases) is to use v.net and v.db.select / v.to.db (from within R) in order to collect attributes
of nodes and edges into R objects (python arrays would work equally well I guess) and then continued working without the geometries. After all operations are done, results are written back to GRASS / SQLite. For parallelization I used doMC package in R and
I was quite satisfied with the performance.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">Kind regards,<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">Stefan<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">P.S.: I also used that kind of approach in the r.connectivity.network addon (GRASS 6).<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif"> grass-user-bounces@lists.osgeo.org [mailto:grass-user-bounces@lists.osgeo.org]
<b>On Behalf Of </b>Mark Wynter<br>
<b>Sent:</b> 13. februar 2015 08:40<br>
<b>To:</b> grass-user@lists.osgeo.org<br>
<b>Subject:</b> [GRASS-user] v.net parallelisation issues<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">I’ve encountered a bottleneck somewhere with <a href="http://v.net">
v.net</a> when scaling out with GNU Parallel… not sure if its an underlying issue with
<a href="http://v.net">v.net</a> or the way I’m calling the batch jobs?<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">I’ve got 32 CPUs and commensurate RAM. What I’m observing is
<a href="http://v.net">v.net</a> CPU utilisation dropping off in accordance with number of jobs running.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">I’ve tried launching a single batch job with single mapset, as well as multiple batch jobs each with their own mapset (and database). I’ve tried both PG and sqlite backends. Same issue.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">The script at the bottom describes the approach of launching multiple batch jobs each with their own map set. Executing a single batch job, and then launching parallel within the batch script is much cleaner code - but the results are
no different.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">I feel I’m so close, yet so far at such a critical stage of project delivery.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Hope someone can help<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Kind regards<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Mark<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">RESULTS<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:8.5pt;font-family:"Menlo",serif">ONE JOB</span><o:p></o:p></p>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:8.5pt;font-family:"Menlo",serif">TOTAL SCRIPT TIME: 70<o:p></o:p></span></p>
</div>
</div>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<div>
<p class="MsoNormal" style="background:black"><span style="font-size:8.5pt;font-family:"Menlo",serif;color:white"> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:8.5pt;font-family:"Menlo",serif">31313 root 20 0 28876 4080 1284 S 76.5 0.0 0:20.25 sqlite <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:8.5pt;font-family:"Menlo",serif">31293 root 20 0 276m 134m 8320 S 68.5 0.2 0:20.22 v.net.distance <o:p></o:p></span></p>
</div>
</div>
<div>
<p class="MsoNormal">—————————<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:8.5pt;font-family:"Menlo",serif">TWO JOBS</span><o:p></o:p></p>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:8.5pt;font-family:"Menlo",serif">TOTAL SCRIPT TIME: 96<o:p></o:p></span></p>
</div>
</div>
<div>
<p class="MsoNormal"><span style="font-size:8.5pt;font-family:"Menlo",serif"><o:p> </o:p></span></p>
</div>
<div>
<div>
<p class="MsoNormal" style="background:black"><span style="font-size:8.5pt;font-family:"Menlo",serif;color:white"> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:8.5pt;font-family:"Menlo",serif">21391 root 20 0 28876 4080 1284 R 53.0 0.0 0:01.90 sqlite <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:8.5pt;font-family:"Menlo",serif">21392 root 20 0 28876 4080 1284 R 52.6 0.0 0:01.86 sqlite <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:8.5pt;font-family:"Menlo",serif">21380 root 20 0 276m 128m 8320 R 49.3 0.2 0:04.02 v.net.distance <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:8.5pt;font-family:"Menlo",serif">21381 root 20 0 276m 128m 8320 S 48.3 0.2 0:03.97 v.net.distance<o:p></o:p></span></p>
</div>
</div>
<div>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Georgia",serif">—————————</span><span style="font-size:8.5pt;font-family:"Menlo",serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:8.5pt;font-family:"Menlo",serif"><o:p> </o:p></span></p>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:8.5pt;font-family:"Menlo",serif">FOUR JOBS</span><span style="font-size:9.0pt;font-family:"Georgia",serif"><o:p></o:p></span></p>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:8.5pt;font-family:"Menlo",serif">TOTAL SCRIPT TIME: 187<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:8.5pt;font-family:"Menlo",serif"><o:p> </o:p></span></p>
</div>
</div>
</div>
<div>
<div>
<p class="MsoNormal" style="background:black"><span style="font-size:8.5pt;font-family:"Menlo",serif;color:white"> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:8.5pt;font-family:"Menlo",serif"> 6953 mark 20 0 180m 100m 9520 S 63.6 0.2 1:47.39 x2goagent <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:8.5pt;font-family:"Menlo",serif">23025 root 20 0 28876 4080 1284 S 21.5 0.0 0:02.03 sqlite <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:8.5pt;font-family:"Menlo",serif">23026 root 20 0 28876 4080 1284 R 19.9 0.0 0:02.08 sqlite <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:8.5pt;font-family:"Menlo",serif">23027 root 20 0 28876 4080 1284 S 19.5 0.0 0:01.87 sqlite <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:8.5pt;font-family:"Menlo",serif">23028 root 20 0 28876 4080 1284 S 19.5 0.0 0:01.84 sqlite <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:8.5pt;font-family:"Menlo",serif">23014 root 20 0 276m 128m 8320 R 18.5 0.2 0:04.06 v.net.distance <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:8.5pt;font-family:"Menlo",serif">23012 root 20 0 276m 128m 8320 R 17.5 0.2 0:03.91 v.net.distance <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:8.5pt;font-family:"Menlo",serif">23011 root 20 0 276m 128m 8320 S 16.9 0.2 0:04.13 v.net.distance <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:8.5pt;font-family:"Menlo",serif">23015 root 20 0 276m 128m 8320 R 16.9 0.2 0:03.80 v.net.distance <o:p></o:p></span></p>
</div>
</div>
<div>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Georgia",serif">—————————</span><span style="font-size:8.5pt;font-family:"Menlo",serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:8.5pt;font-family:"Menlo",serif"><o:p> </o:p></span></p>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:8.5pt;font-family:"Menlo",serif">EIGHT JOBS</span><span style="font-size:9.0pt;font-family:"Georgia",serif"><o:p></o:p></span></p>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:8.5pt;font-family:"Menlo",serif">TOTAL SCRIPT TIME: 373<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:8.5pt;font-family:"Menlo",serif"><o:p> </o:p></span></p>
</div>
<div>
<div>
<p class="MsoNormal" style="background:black"><span style="font-size:8.5pt;font-family:"Menlo",serif;color:white"> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:8.5pt;font-family:"Menlo",serif">27157 root 20 0 28876 4088 1284 S 19.5 0.0 0:42.39 sqlite <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:8.5pt;font-family:"Menlo",serif">27162 root 20 0 28876 4088 1284 R 16.9 0.0 0:40.60 sqlite <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:8.5pt;font-family:"Menlo",serif"> 6953 mark 20 0 181m 101m 9520 S 16.5 0.2 2:18.86 x2goagent <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:8.5pt;font-family:"Menlo",serif">27154 root 20 0 28876 4088 1284 S 16.5 0.0 0:39.38 sqlite <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:8.5pt;font-family:"Menlo",serif">27153 root 20 0 28876 4088 1284 S 16.2 0.0 0:35.60 sqlite <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:8.5pt;font-family:"Menlo",serif">27156 root 20 0 28876 4088 1284 R 16.2 0.0 0:38.18 sqlite <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:8.5pt;font-family:"Menlo",serif">27161 root 20 0 28876 4088 1284 S 15.9 0.0 0:40.96 sqlite <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:8.5pt;font-family:"Menlo",serif">27155 root 20 0 28876 4088 1284 S 15.6 0.0 0:38.41 sqlite <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:8.5pt;font-family:"Menlo",serif">27104 root 20 0 284m 139m 8332 S 14.9 0.2 0:39.94 v.net.distance <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:8.5pt;font-family:"Menlo",serif">27158 root 20 0 28876 4088 1284 R 14.6 0.0 0:37.49 sqlite <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:8.5pt;font-family:"Menlo",serif">27095 root 20 0 284m 138m 8332 S 14.2 0.2 0:34.48 v.net.distance <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:8.5pt;font-family:"Menlo",serif">27099 root 20 0 284m 138m 8332 S 14.2 0.2 0:38.27 v.net.distance <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:8.5pt;font-family:"Menlo",serif">27101 root 20 0 284m 139m 8332 R 14.2 0.2 0:38.80 v.net.distance <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:8.5pt;font-family:"Menlo",serif">27105 root 20 0 284m 139m 8332 R 14.2 0.2 0:37.95 v.net.distance <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:8.5pt;font-family:"Menlo",serif">27093 root 20 0 284m 138m 8332 R 13.9 0.2 0:32.64 v.net.distance <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:8.5pt;font-family:"Menlo",serif">27102 root 20 0 284m 140m 8332 R 13.6 0.2 0:40.90 v.net.distance <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:8.5pt;font-family:"Menlo",serif">27094 root 20 0 284m 138m 8332 R 13.2 0.2 0:35.78 v.net.distance <o:p></o:p></span></p>
</div>
</div>
</div>
</div>
<div>
<p class="MsoNormal">—————————<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<div>
<p class="MsoNormal">################################################<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">############ WORKER FUNCTION ############# <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">################################################<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"># CREATE MAPSETS AND BASH SCRIPTS FOR EACH CPU<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">fn_worker (){<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">#######################<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"># copy mapset<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">#######################<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">cp -R /var/tmp/jtw/PERMANENT /var/tmp/jtw/batch_"$1"<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">#######################<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"># generate batch_job file<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">#######################<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">echo -e '#!/bin/bash<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">dbsettings="/mnt/data/common/repos/cf_private/settings/current.sh"<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">source $dbsettings<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">cpu='$1'<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">jid=`psql -d $dbname -U $username -A -t -c "SELECT min(jid) FROM jtw.nsw_tz_joblist WHERE processed = false and cpu = '$1';"`<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">o_tz11=`psql -d $dbname -U $username -A -t -c "SELECT o_tz11 FROM jtw.nsw_tz_joblist WHERE jid = $jid;"`<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">o_cat=`psql -d $dbname -U $username -A -t -c "SELECT o_tz11 FROM jtw.nsw_tz_joblist WHERE jid = $jid;"`<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">d_cat=`psql -d $dbname -U $username -A -t -c "SELECT d_tz11 FROM jtw.nsw_tz_joblist WHERE jid = $jid;"`<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">layername="temp_"$jid<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">v.net.distance --overwrite in=nsw_road_network_final_connected@batch_'$1' out=$layername from_layer=2 to_layer=2 from_cats=$d_cat to_cats=$o_cat arc_column=fwdcost arc_backward_column=bwdcost<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">v.out.ogr --o input=$layername output=/var/tmp/$layername type=line<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">ogr2ogr -overwrite -f "PostgreSQL" PG:"host=localhost dbname=o$dbname user=$username password=$password" /var/tmp/$layername/$layername.shp -nln jtw.$layername -s_srs EPSG:3577 -t_srs EPSG:3577 -a_srs EPSG:3577 -nlt LINESTRING<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">psql -d $dbname -U $username -c "INSERT INTO jtw.nsw_tz_journey_paths<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">With s AS (SELECT a.cat, a.tcat, b.tz_code11 as o_tz11, c.tz_code11 as d_tz11, d.lid, d.wkb_geometry, e.employed_persons FROM jtw.$layername a, grass.nsw_tz_centroids_nodes b, grass.nsw_tz_centroids_nodes c, jtw.nsw_road_network_final_net_att
d, jtw.nsw_tz_volumes e WHERE a.tcat = b.cat AND a.cat = c.cat AND ST_Equals(a.wkb_geometry, d.wkb_geometry) AND d.type <> '\'service_line\'' AND b.tz_code11 = e.o_tz11 AND c.tz_code11 = e.d_tz11 AND e.mode9 = 4) SELECT NEXTVAL('\'jtw.nsw_tz_journey_paths_jid_seq\''),
o_tz11, d_tz11, lid, wkb_geometry, employed_persons FROM s; UPDATE jtw.nsw_tz_joblist SET processed = true WHERE jid = $jid;"<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">#end of job file' > /var/tmp/jtw/jobs/batch_$1.sh<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">#######################<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">chmod u+x /var/tmp/jtw/jobs/batch_$1.sh<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">}<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">export -f fn_worker<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"># remove previous mapsets before writing new files<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">rm -rf /var/tmp/jtw/batch*<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">rm -rf /var/tmp/jtw/jobs/batch*<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">#execute in parallel<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">seq 1 4 | parallel fn_worker {1}<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">wait<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">#######################<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">################################################<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">####### JOB SCHEDULER ######## <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">################################################<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">#\\\\\\\\\\\\\\\\\\\\\\\\\<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">START_T1=$(date +%s)<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">#\\\\\\\\\\\\\\\\\\\\\\\\\<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">fn_worker (){<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">export GRASS_BATCH_JOB=/var/tmp/jtw/jobs/batch_$1.sh<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">grass70 /var/tmp/jtw/batch_$1<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">unset GRASS_BATCH_JOB<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">}<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">export -f fn_worker<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">seq 1 4 | parallel fn_worker {1}<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">wait<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">#\\\\\\\\\\\\\\\\\\\\\\\\\<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">END_T1=$(date +%s)<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">#\\\\\\\\\\\\\\\\\\\\\\\\\<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">TOTAL_DIFF=$(( $END_T1 - $START_T1 ))<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">echo "TOTAL SCRIPT TIME: $TOTAL_DIFF"<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">#\\\\\\\\\\\\\\\\\\\\\\\\\<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">################################################<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><br>
The slow rate of writing out the v.net.allpair results from<br>
PostgreSQL was due to the sheer volume of line strings, as the number<br>
of pairs increased (n^2). Simple math said stop. I?ve since<br>
changed my approach and am using v.net.distance in a novel way where<br>
the to_cat is the origin, and the from_cat is a string of<br>
destinations - this is an equivalent way of generating multiple<br>
v.net.paths in a single operation. Moreover, I?m feeding each origin<br>
- destination collection into GNU Parallel as a separate job, so it<br>
rips through the data at scale!<o:p></o:p></p>
</blockquote>
<p class="MsoNormal"><o:p> </o:p></p>
</blockquote>
</div>
</div>
</body>
</html>