[SAC] [OSGeo] #1325: ProjectsVM not responding
OSGeo
trac_osgeo at osgeo.org
Thu Apr 10 07:49:58 PDT 2014
#1325: ProjectsVM not responding
---------------------------+------------------------------------------------
Reporter: jmckenna | Owner: sac@…
Type: task | Status: new
Priority: critical | Milestone:
Component: Systems Admin | Keywords:
---------------------------+------------------------------------------------
Comment(by crschmidt):
I don't know the current state of this, but here's some things I do know:
1. The ProjectsVM Apache was swamped this morning.
2. Somehow, in the past week, the number of available connections to the
projectsVM apache was dropped:
http://webextra.osgeo.osuosl.org/munin/osgeo.org/projects.osgeo.org/apache_processes.html
3. Markus raised that back up.
4. As always, a huge number of botnets were hitting the OSGeo projects VM
trying to use it as an open proxy. (These requests were just returning
404s.) I blackholed a number of IPs to pull the number of incoming
connections down; I think I got about 40% of the incoming spam
connections. iptables --list will show the ranges I blacklisted; I tried
to be relatively conservative. I used the following command to find IPs to
blacklist.
sudo tail -n 100 /var/log/apache2/docs.geotools.org-access_log | grep
"http:" | cut -f 1-3 -d'.' | sort | uniq -c | sort -n
Now that apache is configured correctly and the incoming spam is decreased
a bit, the disks are still massively underperforming; iostat -x -m 2 shows
that there is as little as 2-3 disk seeks/sec going on with full
utilization and high await times. This usually means bad things; If the
raid batteries are not replaced yet, this would be consistent with
WriteThrough mode.
This means that the high load average on the projects vm appears to be
entirely due to bad disk performance. I don't know how to debug more --
the machine seems otherwise fine -- and I think the machine will run, with
abnormally high load numbers, though access that requires disk seeks will
be much slower than usual.
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz
avgqu-sz await svctm %util
vda 0.00 0.00 0.00 0.00 0.00 0.00 0.00
1.00 0.00 0.00 100.00
--
Ticket URL: <http://trac.osgeo.org/osgeo/ticket/1325#comment:11>
OSGeo <http://www.osgeo.org/>
OSGeo committee and general foundation issue tracker.
More information about the Sac
mailing list