[SAC] [OSGeo] #1325: ProjectsVM not responding

OSGeo trac_osgeo at osgeo.org
Thu Apr 10 07:49:58 PDT 2014


#1325: ProjectsVM not responding
---------------------------+------------------------------------------------
 Reporter:  jmckenna       |       Owner:  sac@…              
     Type:  task           |      Status:  new                
 Priority:  critical       |   Milestone:                     
Component:  Systems Admin  |    Keywords:                     
---------------------------+------------------------------------------------

Comment(by crschmidt):

 I don't know the current state of this, but here's some things I do know:

  1. The ProjectsVM Apache was swamped this morning.
  2. Somehow, in the past week, the number of available connections to the
 projectsVM apache was dropped:
 http://webextra.osgeo.osuosl.org/munin/osgeo.org/projects.osgeo.org/apache_processes.html
  3. Markus raised that back up.
  4. As always, a huge number of botnets were hitting the OSGeo projects VM
 trying to use it as an open proxy. (These requests were just returning
 404s.) I blackholed a number of IPs to pull the number of incoming
 connections down; I think I got about 40% of the incoming spam
 connections. iptables --list will show the ranges I blacklisted; I tried
 to be relatively conservative. I used the following command to find IPs to
 blacklist.

 sudo tail -n 100 /var/log/apache2/docs.geotools.org-access_log | grep
 "http:" | cut -f 1-3 -d'.' | sort | uniq -c | sort -n

 Now that apache is configured correctly and the incoming spam is decreased
 a bit, the disks are still massively underperforming; iostat -x -m 2 shows
 that there is as little as 2-3 disk seeks/sec going on with full
 utilization and high await times. This usually means bad things; If the
 raid batteries are not replaced yet, this would be consistent with
 WriteThrough mode.

 This means that the high load average on the projects vm appears to be
 entirely due to bad disk performance. I don't know how to debug more --
 the machine seems otherwise fine -- and I think the machine will run, with
 abnormally high load numbers, though access that requires disk seeks will
 be much slower than usual.

 Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s avgrq-sz
 avgqu-sz   await  svctm  %util
 vda               0.00     0.00    0.00    0.00     0.00     0.00     0.00
 1.00    0.00   0.00 100.00

-- 
Ticket URL: <http://trac.osgeo.org/osgeo/ticket/1325#comment:11>
OSGeo <http://www.osgeo.org/>
OSGeo committee and general foundation issue tracker.


More information about the Sac mailing list