[GRASS-user] v.net parallelisation issues
Mark Wynter
mark at dimensionaledge.com
Fri Feb 13 17:14:39 PST 2015
Hi Moritz, Stefan (and Marcus if you’re around?)
Every day is a new day.
Not IOPs, Not memory, Not CPU… Hmm, how about reboot...
>>
>> As you have suspected, I get no benefit from additional CPUs.
>
> Are you sure the problem is CPU-bound ?
I started by testing v.net (the maintenance module) in parallel - and observed pretty much linear scaling in performance over and above 2 parallel jobs.
So I then proceeded to test v.net.distance… and sure enough, it is now scaling. Really different looking CPU performance profiles to yesterday, when it wasn’t scaling. On top, all the v.net CPUs are humming at 50%, the other 50% used by pg processes.
Below are some iostat profiles of the different parallel test.
So v.net.distance does parallelise nicely. The neatest way is to launch GNU parallel from within a single grass_batch_job. I will write up on the grass wiki in the coming 2 weeks.
Mark
:-)))))
---------------V.NET ---------------
Single job
v.net
iostat 1
TOTAL SCRIPT TIME: 284
avg-cpu: %user %nice %system %iowait %steal %idle
15.92 0.00 35.32 0.50 0.00 48.26
Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
xvde 18.00 8.00 744.00 8 744
xvdj 0.00 0.00 0.00 0 0
avg-cpu: %user %nice %system %iowait %steal %idle
20.00 0.00 34.00 0.50 0.00 45.50
Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
xvde 9.90 0.00 633.66 0 640
xvdj 0.00 0.00 0.00 0 0
avg-cpu: %user %nice %system %iowait %steal %idle
22.50 0.00 35.50 0.50 0.00 41.50
Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
xvde 9.00 0.00 592.00 0 592
xvdj 0.00 0.00 0.00 0 0
Two parallel jobs
v.net
iostat 1
TOTAL SCRIPT TIME: 397
avg-cpu: %user %nice %system %iowait %steal %idle
32.16 0.00 67.84 0.00 0.00 0.00
Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
xvde 19.00 0.00 1024.00 0 1024
xvdj 0.00 0.00 0.00 0 0
avg-cpu: %user %nice %system %iowait %steal %idle
39.50 0.00 60.50 0.00 0.00 0.00
Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
xvde 19.00 0.00 912.00 0 912
xvdj 0.00 0.00 0.00 0 0
avg-cpu: %user %nice %system %iowait %steal %idle
35.50 0.00 64.50 0.00 0.00 0.00
Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
xvde 16.00 0.00 1024.00 0 1024
xvdj 0.00 0.00 0.00 0 0
Four parallel jobs
v.net
iostat 1
TOTAL SCRIPT TIME: 388
avg-cpu: %user %nice %system %iowait %steal %idle
32.92 0.00 67.08 0.00 0.00 0.00
Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
xvde 31.00 0.00 1952.00 0 1952
xvdj 0.00 0.00 0.00 0 0
avg-cpu: %user %nice %system %iowait %steal %idle
33.83 0.00 66.17 0.00 0.00 0.00
Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
xvde 35.00 0.00 2016.00 0 2016
xvdj 0.00 0.00 0.00 0 0
avg-cpu: %user %nice %system %iowait %steal %idle
32.92 0.00 67.08 0.00 0.00 0.00
Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
xvde 61.00 0.00 4032.00 0 4032
xvdj 0.00 0.00 0.00 0 0
---------------V.NET.DISTANCE ---------------
Single job
v.net.distance
iostat 1
TOTAL SCRIPT TIME: 88
avg-cpu: %user %nice %system %iowait %steal %idle
4.00 0.00 21.00 0.00 0.00 75.00
Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
xvde 0.00 0.00 0.00 0 0
xvdj 0.00 0.00 0.00 0 0
avg-cpu: %user %nice %system %iowait %steal %idle
7.75 0.00 21.75 0.00 0.00 70.50
Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
xvde 0.00 0.00 0.00 0 0
xvdj 0.00 0.00 0.00 0 0
avg-cpu: %user %nice %system %iowait %steal %idle
6.27 0.00 22.56 0.00 0.00 71.18
Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
xvde 4.00 0.00 56.00 0 56
xvdj 0.00 0.00 0.00 0 0
Two parallel jobs
v.net.distance
iostat 1
TOTAL SCRIPT TIME: 96
avg-cpu: %user %nice %system %iowait %steal %idle
9.02 0.00 40.85 0.00 0.00 50.13
Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
xvde 0.00 0.00 0.00 0 0
xvdj 0.00 0.00 0.00 0 0
avg-cpu: %user %nice %system %iowait %steal %idle
11.75 0.00 43.75 0.00 0.00 44.50
Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
xvde 3.00 0.00 40.00 0 40
xvdj 0.00 0.00 0.00 0 0
avg-cpu: %user %nice %system %iowait %steal %idle
9.50 0.00 40.75 0.00 0.00 49.75
Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
xvde 0.00 0.00 0.00 0 0
xvdj 0.00 0.00 0.00 0 0
Four parallel jobs
v.net.distance
iostat 1
TOTAL SCRIPT TIME: 124
avg-cpu: %user %nice %system %iowait %steal %idle
17.96 0.00 82.04 0.00 0.00 0.00
Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
xvde 3.00 0.00 32.00 0 32
xvdj 0.00 0.00 0.00 0 0
avg-cpu: %user %nice %system %iowait %steal %idle
18.50 0.00 81.50 0.00 0.00 0.00
Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
xvde 0.00 0.00 0.00 0 0
xvdj 0.00 0.00 0.00 0 0
avg-cpu: %user %nice %system %iowait %steal %idle
21.50 0.00 78.50 0.00 0.00 0.00
Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
xvde 0.00 0.00 0.00 0 0
xvdj 7.00 0.00 64.00 0 64
Eight parallel jobs
v.net.distance
iostat 1
TOTAL SCRIPT TIME: 122
avg-cpu: %user %nice %system %iowait %steal %idle
19.00 0.00 81.00 0.00 0.00 0.00
Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
xvde 0.00 0.00 0.00 0 0
xvdj 0.00 0.00 0.00 0 0
avg-cpu: %user %nice %system %iowait %steal %idle
19.50 0.00 80.50 0.00 0.00 0.00
Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
xvde 0.00 0.00 0.00 0 0
xvdj 0.00 0.00 0.00 0 0
avg-cpu: %user %nice %system %iowait %steal %idle
18.65 0.00 81.35 0.00 0.00 0.00
Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
xvde 3.00 0.00 48.00 0 48
xvdj 0.00 0.00 0.00 0 0
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/grass-user/attachments/20150214/e1667718/attachment-0001.html>
More information about the grass-user
mailing list