<html><head><meta http-equiv="Content-Type" content="text/html charset=windows-1252"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;"><div>Ive encountered a bottleneck somewhere with <a href="http://v.net">v.net</a> when scaling out with GNU Parallel
not sure if its an underlying issue with <a href="http://v.net">v.net</a> or the way Im calling the batch jobs?</div><div><br></div><div>Ive got 32 CPUs and commensurate RAM. What Im observing is <a href="http://v.net">v.net</a> CPU utilisation dropping off in accordance with number of jobs running.</div><div><br></div><div>Ive tried launching a single batch job with single mapset, as well as multiple batch jobs each with their own mapset (and database). Ive tried both PG and sqlite backends. Same issue.</div><div><br></div><div>The script at the bottom describes the approach of launching multiple batch jobs each with their own map set. Executing a single batch job, and then launching parallel within the batch script is much cleaner code - but the results are no different.</div><div><br></div><div>I feel Im so close, yet so far at such a critical stage of project delivery.</div><div><br></div><div>Hope someone can help</div><div><br></div><div>Kind regards</div><div>Mark</div><div><br></div><div>RESULTS</div><div><br></div><div><div><span style="font-family: Menlo; font-size: 11px;">ONE JOB</span></div><div><div style="margin: 0px; font-size: 11px; font-family: Menlo;">TOTAL SCRIPT TIME: 70</div></div></div><div><br></div><div><div style="margin: 0px; font-size: 11px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND </div><div style="margin: 0px; font-size: 11px; font-family: Menlo;">31313 root 20 0 28876 4080 1284 S 76.5 0.0 0:20.25 sqlite </div><div style="margin: 0px; font-size: 11px; font-family: Menlo;">31293 root 20 0 276m 134m 8320 S 68.5 0.2 0:20.22 v.net.distance </div></div><div></div><div><br></div><div><span style="font-family: Menlo; font-size: 11px;">TWO JOBS</span></div><div><div style="margin: 0px; font-size: 11px; font-family: Menlo;">TOTAL SCRIPT TIME: 96</div></div><div style="margin: 0px; font-size: 11px; font-family: Menlo;"><br></div><div style="margin: 0px; font-size: 11px; font-family: Menlo;"><div style="margin: 0px; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND </div><div style="margin: 0px;">21391 root 20 0 28876 4080 1284 R 53.0 0.0 0:01.90 sqlite </div><div style="margin: 0px;">21392 root 20 0 28876 4080 1284 R 52.6 0.0 0:01.86 sqlite </div><div style="margin: 0px;">21380 root 20 0 276m 128m 8320 R 49.3 0.2 0:04.02 v.net.distance </div><div style="margin: 0px;">21381 root 20 0 276m 128m 8320 S 48.3 0.2 0:03.97 v.net.distance</div></div><div style="margin: 0px; font-size: 11px; font-family: Menlo;"><span style="font-family: Georgia; font-size: 12px;"></span></div><div style="margin: 0px; font-size: 11px; font-family: Menlo;"><br></div><div style="margin: 0px; font-size: 11px; font-family: Menlo;"><div style="font-family: Georgia; font-size: 12px;"><span style="font-family: Menlo; font-size: 11px;">FOUR JOBS</span></div><div style="font-family: Georgia; font-size: 12px;"><div style="margin: 0px; font-size: 11px; font-family: Menlo;">TOTAL SCRIPT TIME: 187</div><div style="margin: 0px; font-size: 11px; font-family: Menlo;"><br></div></div></div><div style="margin: 0px; font-size: 11px; font-family: Menlo;"><div style="margin: 0px; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND </div><div style="margin: 0px;"> 6953 mark 20 0 180m 100m 9520 S 63.6 0.2 1:47.39 x2goagent </div><div style="margin: 0px;">23025 root 20 0 28876 4080 1284 S 21.5 0.0 0:02.03 sqlite </div><div style="margin: 0px;">23026 root 20 0 28876 4080 1284 R 19.9 0.0 0:02.08 sqlite </div><div style="margin: 0px;">23027 root 20 0 28876 4080 1284 S 19.5 0.0 0:01.87 sqlite </div><div style="margin: 0px;">23028 root 20 0 28876 4080 1284 S 19.5 0.0 0:01.84 sqlite </div><div style="margin: 0px;">23014 root 20 0 276m 128m 8320 R 18.5 0.2 0:04.06 v.net.distance </div><div style="margin: 0px;">23012 root 20 0 276m 128m 8320 R 17.5 0.2 0:03.91 v.net.distance </div><div style="margin: 0px;">23011 root 20 0 276m 128m 8320 S 16.9 0.2 0:04.13 v.net.distance </div><div style="margin: 0px;">23015 root 20 0 276m 128m 8320 R 16.9 0.2 0:03.80 v.net.distance </div></div><div style="margin: 0px; font-size: 11px; font-family: Menlo;"><span style="font-family: Georgia; font-size: 12px;"></span></div><div style="margin: 0px; font-size: 11px; font-family: Menlo;"><span style="font-family: Georgia; font-size: 12px;"><br></span></div><div style="margin: 0px; font-size: 11px; font-family: Menlo;"><div style="font-family: Georgia; font-size: 12px;"><span style="font-family: Menlo; font-size: 11px;">EIGHT JOBS</span></div><div style="font-family: Georgia; font-size: 12px;"><div style="margin: 0px; font-size: 11px; font-family: Menlo;">TOTAL SCRIPT TIME: 373</div><div style="margin: 0px; font-size: 11px; font-family: Menlo;"><br></div><div style="margin: 0px; font-size: 11px; font-family: Menlo;"><div style="margin: 0px; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND </div><div style="margin: 0px;">27157 root 20 0 28876 4088 1284 S 19.5 0.0 0:42.39 sqlite </div><div style="margin: 0px;">27162 root 20 0 28876 4088 1284 R 16.9 0.0 0:40.60 sqlite </div><div style="margin: 0px;"> 6953 mark 20 0 181m 101m 9520 S 16.5 0.2 2:18.86 x2goagent </div><div style="margin: 0px;">27154 root 20 0 28876 4088 1284 S 16.5 0.0 0:39.38 sqlite </div><div style="margin: 0px;">27153 root 20 0 28876 4088 1284 S 16.2 0.0 0:35.60 sqlite </div><div style="margin: 0px;">27156 root 20 0 28876 4088 1284 R 16.2 0.0 0:38.18 sqlite </div><div style="margin: 0px;">27161 root 20 0 28876 4088 1284 S 15.9 0.0 0:40.96 sqlite </div><div style="margin: 0px;">27155 root 20 0 28876 4088 1284 S 15.6 0.0 0:38.41 sqlite </div><div style="margin: 0px;">27104 root 20 0 284m 139m 8332 S 14.9 0.2 0:39.94 v.net.distance </div><div style="margin: 0px;">27158 root 20 0 28876 4088 1284 R 14.6 0.0 0:37.49 sqlite </div><div style="margin: 0px;">27095 root 20 0 284m 138m 8332 S 14.2 0.2 0:34.48 v.net.distance </div><div style="margin: 0px;">27099 root 20 0 284m 138m 8332 S 14.2 0.2 0:38.27 v.net.distance </div><div style="margin: 0px;">27101 root 20 0 284m 139m 8332 R 14.2 0.2 0:38.80 v.net.distance </div><div style="margin: 0px;">27105 root 20 0 284m 139m 8332 R 14.2 0.2 0:37.95 v.net.distance </div><div style="margin: 0px;">27093 root 20 0 284m 138m 8332 R 13.9 0.2 0:32.64 v.net.distance </div><div style="margin: 0px;">27102 root 20 0 284m 140m 8332 R 13.6 0.2 0:40.90 v.net.distance </div><div style="margin: 0px;">27094 root 20 0 284m 138m 8332 R 13.2 0.2 0:35.78 v.net.distance </div></div></div></div><div></div><div><br></div><div><div>################################################</div><div>############ WORKER FUNCTION ############# </div><div>################################################</div><div># CREATE MAPSETS AND BASH SCRIPTS FOR EACH CPU</div><div>fn_worker (){</div><div><br></div><div>#######################</div><div># copy mapset</div><div>#######################</div><div>cp -R /var/tmp/jtw/PERMANENT /var/tmp/jtw/batch_"$1"</div><div><br></div><div>#######################</div><div># generate batch_job file</div><div>#######################</div><div>echo -e '#!/bin/bash</div><div>dbsettings="/mnt/data/common/repos/cf_private/settings/current.sh"</div><div>source $dbsettings</div><div>cpu='$1'</div><div><br></div><div>jid=`psql -d $dbname -U $username -A -t -c "SELECT min(jid) FROM jtw.nsw_tz_joblist WHERE processed = false and cpu = '$1';"`</div><div>o_tz11=`psql -d $dbname -U $username -A -t -c "SELECT o_tz11 FROM jtw.nsw_tz_joblist WHERE jid = $jid;"`</div><div>o_cat=`psql -d $dbname -U $username -A -t -c "SELECT o_tz11 FROM jtw.nsw_tz_joblist WHERE jid = $jid;"`</div><div>d_cat=`psql -d $dbname -U $username -A -t -c "SELECT d_tz11 FROM jtw.nsw_tz_joblist WHERE jid = $jid;"`</div><div>layername="temp_"$jid</div><div><br></div><div>v.net.distance --overwrite in=nsw_road_network_final_connected@batch_'$1' out=$layername from_layer=2 to_layer=2 from_cats=$d_cat to_cats=$o_cat arc_column=fwdcost arc_backward_column=bwdcost</div><div><br></div><div>v.out.ogr --o input=$layername output=/var/tmp/$layername type=line</div><div><br></div><div>ogr2ogr -overwrite -f "PostgreSQL" PG:"host=localhost dbname=o$dbname user=$username password=$password" /var/tmp/$layername/$layername.shp -nln jtw.$layername -s_srs EPSG:3577 -t_srs EPSG:3577 -a_srs EPSG:3577 -nlt LINESTRING</div><div><br></div><div>psql -d $dbname -U $username -c "INSERT INTO jtw.nsw_tz_journey_paths</div><div>With s AS (SELECT a.cat, a.tcat, b.tz_code11 as o_tz11, c.tz_code11 as d_tz11, d.lid, d.wkb_geometry, e.employed_persons FROM jtw.$layername a, grass.nsw_tz_centroids_nodes b, grass.nsw_tz_centroids_nodes c, jtw.nsw_road_network_final_net_att d, jtw.nsw_tz_volumes e WHERE a.tcat = b.cat AND a.cat = c.cat AND ST_Equals(a.wkb_geometry, d.wkb_geometry) AND d.type <> '\'service_line\'' AND b.tz_code11 = e.o_tz11 AND c.tz_code11 = e.d_tz11 AND e.mode9 = 4) SELECT NEXTVAL('\'jtw.nsw_tz_journey_paths_jid_seq\''), o_tz11, d_tz11, lid, wkb_geometry, employed_persons FROM s; UPDATE jtw.nsw_tz_joblist SET processed = true WHERE jid = $jid;"</div><div><br></div><div>#end of job file' > /var/tmp/jtw/jobs/batch_$1.sh</div><div>#######################</div><div><br></div><div>chmod u+x /var/tmp/jtw/jobs/batch_$1.sh</div><div>}</div><div>export -f fn_worker</div><div><br></div><div># remove previous mapsets before writing new files</div><div>rm -rf /var/tmp/jtw/batch*</div><div>rm -rf /var/tmp/jtw/jobs/batch*</div><div><br></div><div>#execute in parallel</div><div>seq 1 4 | parallel fn_worker {1}</div><div>wait</div><div>#######################</div><div><br></div><div><br></div><div>################################################</div><div>####### JOB SCHEDULER ######## </div><div>################################################</div><div><br></div><div>#\\\\\\\\\\\\\\\\\\\\\\\\\</div><div>START_T1=$(date +%s)</div><div>#\\\\\\\\\\\\\\\\\\\\\\\\\</div><div><br></div><div>fn_worker (){</div><div>export GRASS_BATCH_JOB=/var/tmp/jtw/jobs/batch_$1.sh</div><div>grass70 /var/tmp/jtw/batch_$1</div><div>unset GRASS_BATCH_JOB</div><div>}</div><div>export -f fn_worker</div><div><br></div><div>seq 1 4 | parallel fn_worker {1}</div><div>wait</div><div><br></div><div>#\\\\\\\\\\\\\\\\\\\\\\\\\</div><div>END_T1=$(date +%s)</div><div>#\\\\\\\\\\\\\\\\\\\\\\\\\</div><div>TOTAL_DIFF=$(( $END_T1 - $START_T1 ))</div><div>echo "TOTAL SCRIPT TIME: $TOTAL_DIFF"</div><div>#\\\\\\\\\\\\\\\\\\\\\\\\\</div><div><br></div><div>################################################</div><div><br></div></div><br><div><blockquote type="cite"><blockquote type="cite"><br>The slow rate of writing out the v.net.allpair results from<br>PostgreSQL was due to the sheer volume of line strings, as the number<br>of pairs increased (n^2). Simple math said stop. I?ve since<br>changed my approach and am using v.net.distance in a novel way where<br>the to_cat is the origin, and the from_cat is a string of<br>destinations - this is an equivalent way of generating multiple<br>v.net.paths in a single operation. Moreover, I?m feeding each origin<br>- destination collection into GNU Parallel as a separate job, so it<br>rips through the data at scale!<br></blockquote><br></blockquote></div></body></html>