[SAC] dsmc - tivoli backup system on osgeo1
Frank Warmerdam
warmerdam at pobox.com
Tue Nov 24 02:46:34 EST 2009
Martin Spott wrote:
> On Thu, Nov 19, 2009 at 04:36:12PM -0500, Frank Warmerdam wrote:
>
>> I am increasingly convinced that the automated backup done by dsmc
>> on osgeo1 is responsible for the IO contention that leads to service
>> unavailability on a fairly frequent basis.
>
> Today I have installed 'iostat' on the 'osgeo1' machine. The next time
> you suspect IO contention (when I'm not around), please paste the
> output of approx. half a minute into an EMail, running the command as:
>
> # ~> iostat -x 5
Martin,
log for a part of a minute attached. This is during a period when the load
average spiked to 23 or so, and "wait states" were around 65% in the top
report.
In this case, there is no sign of dsmc. The top of the top report looks like:
top - 02:43:22 up 19 days, 5:38, 2 users, load average: 23.77, 22.19, 19.00
Tasks: 333 total, 1 running, 332 sleeping, 0 stopped, 0 zombie
Cpu(s): 14.9% us, 2.6% sy, 0.0% ni, 21.9% id, 60.6% wa, 0.1% hi, 0.0% si
Mem: 2074860k total, 2050372k used, 24488k free, 5060k buffers
Swap: 2040244k total, 723380k used, 1316864k free, 453616k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
26106 apache 17 0 64836 45m 7516 S 25.8 2.2 0:15.01 httpd
26116 postgres 18 0 27032 11m 10m D 6.9 0.6 0:01.22 postmaster
26162 apache 16 0 40060 21m 6248 S 5.9 1.1 0:01.17 httpd
25202 apache 15 0 65620 42m 7656 D 4.6 2.1 0:11.93 httpd
26114 root 16 0 3252 540 188 S 4.6 0.0 0:00.74 gzip
22970 apache 16 0 228m 137m 7636 S 3.6 6.8 1:04.32 httpd
25971 postgres 17 0 26424 11m 9m D 3.6 0.5 0:03.70 postmaster
26149 postgres 16 0 26336 11m 9m S 3.3 0.5 0:01.87 postmaster
2922 mysql 16 0 200m 93m 3344 S 2.6 4.6 3016:12 mysqld
25456 apache 15 0 76080 50m 7776 D 1.0 2.5 0:25.90 httpd
26159 postgres 15 0 26336 11m 10m D 1.0 0.5 0:00.61 postmaster
67 root 15 0 0 0 0 S 0.7 0.0 33:18.59 kswapd0
25937 postgres 15 0 26380 11m 9m S 0.7 0.5 0:02.09 postmaster
26017 postgres 16 0 26768 11m 10m D 0.7 0.6 0:01.40 postmaster
26040 postgres 15 0 26380 11m 9m D 0.7 0.5 0:00.77 postmaster
26113 root 16 0 5020 1676 1296 S 0.7 0.1 0:00.11 pg_dump
26140 root 16 0 3188 1156 772 R 0.7 0.1 0:00.56 top
25252 postgres 16 0 26324 8440 7520 S 0.3 0.4 0:00.15 postmaster
25792 postgres 15 0 26336 10m 10m D 0.3 0.5 0:00.63 postmaster
25846 postgres 15 0 26952 11m 10m D 0.3 0.6 0:02.64 postmaster
25883 postgres 17 0 26336 10m 9512 D 0.3 0.5 0:00.22 postmaster
25993 postgres 15 0 26344 10m 9m D 0.3 0.5 0:00.45 postmaster
26003 postgres 17 0 26336 10m 9.9m D 0.3 0.5 0:00.35 postmaster
26020 apache 16 0 62312 43m 7664 S 0.3 2.2 0:05.19 httpd
26093 postgres 15 0 26336 11m 10m D 0.3 0.5 0:00.79 postmaster
26151 postgres 16 0 26340 8064 7068 D 0.3 0.4 0:00.08 postmaster
Best regards,
--
---------------------------------------+--------------------------------------
I set the clouds in motion - turn up | Frank Warmerdam, warmerdam at pobox.com
light and sound - activate the windows | http://pobox.com/~warmerdam
and watch the world go round - Rush | Geospatial Programmer for Rent
-------------- next part --------------
[root at osgeo1 httpd]# iostat -x 5
Linux 2.6.9-89.0.16.ELsmp (osgeo1.osgeo.org) 11/24/2009
avg-cpu: %user %nice %sys %iowait %idle
34.65 0.02 4.59 9.61 51.14
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
sda 3.12 220.30 74.58 60.67 1721.32 2248.04 860.66 1124.02 29.35 0.46 3.43 2.76 37.39
sda1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 3.68 0.00 16.81 14.17 0.00
sda2 0.63 216.99 73.99 60.41 1696.73 2219.31 848.36 1109.66 29.14 0.20 1.52 2.78 37.38
sda3 2.48 3.31 0.59 0.27 24.59 28.72 12.30 14.36 62.16 0.26 303.44 41.95 3.60
avg-cpu: %user %nice %sys %iowait %idle
8.50 0.00 1.95 66.90 22.65
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
sda 123.65 85.37 322.24 19.64 11137.47 840.08 5568.74 420.04 35.03 72.03 214.20 2.93 100.24
sda1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda2 3.81 85.37 300.40 19.64 10005.61 840.08 5002.81 420.04 33.89 69.33 220.35 3.13 100.24
sda3 119.84 0.00 21.84 0.00 1131.86 0.00 565.93 0.00 51.82 2.69 123.99 45.85 100.16
avg-cpu: %user %nice %sys %iowait %idle
9.55 0.00 2.15 58.55 29.75
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
sda 109.38 153.09 306.39 29.54 8202.79 1461.08 4101.40 730.54 28.77 79.39 233.44 2.97 99.82
sda1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda2 2.00 153.09 283.23 29.54 7160.08 1461.08 3580.04 730.54 27.56 77.77 245.44 3.19 99.82
sda3 107.39 0.00 23.15 0.00 1042.71 0.00 521.36 0.00 45.03 1.63 71.41 42.71 98.88
avg-cpu: %user %nice %sys %iowait %idle
13.60 0.00 1.80 64.40 20.20
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
sda 29.80 66.80 300.40 22.20 6843.20 713.60 3421.60 356.80 23.42 77.56 233.92 3.10 100.04
sda1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda2 2.00 66.80 289.40 22.20 6532.80 713.60 3266.40 356.80 23.26 75.76 236.51 3.21 100.04
sda3 27.80 0.00 11.00 0.00 310.40 0.00 155.20 0.00 28.22 1.80 160.42 76.07 83.68
avg-cpu: %user %nice %sys %iowait %idle
8.65 0.00 1.35 51.87 38.13
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
sda 55.80 45.40 314.00 13.40 7384.00 470.40 3692.00 235.20 23.99 78.62 246.03 3.06 100.04
sda1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda2 2.60 45.40 297.40 13.40 6816.00 470.40 3408.00 235.20 23.44 76.43 253.62 3.22 100.04
sda3 53.20 0.00 16.60 0.00 568.00 0.00 284.00 0.00 34.22 2.20 103.83 56.67 94.08
avg-cpu: %user %nice %sys %iowait %idle
21.35 0.00 5.50 60.10 13.05
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
sda 23.80 239.40 298.20 61.60 8465.60 2462.40 4232.80 1231.20 30.37 88.07 242.82 2.78 100.06
sda1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda2 1.20 199.80 288.40 59.60 8214.40 2112.00 4107.20 1056.00 29.67 86.05 244.13 2.88 100.06
sda3 22.60 39.60 9.80 2.00 251.20 350.40 125.60 175.20 50.98 2.02 204.44 69.51 82.02
More information about the Sac
mailing list