[SAC] System Load Testing

[Frank Warmerdam]

Folks,

I did a bit of load testing on www.osgeo.org which was otherwise very
quiet tonight.

I found:
  o I could pull one drupal web page at about 60pages per second
    (http://www.osgeo.org)

  o I found pulling a non-existant page ran about the same speed.
  o I can pull http://trac.osgeo.org/gdal at about 2400pages/second.
  o I could run a recusive wget over the web site with a reasonably modest
    impact on the server - no pages/second stats though.
  o If I had 10 wgets pulling recursively in parallel the load average was
    driven up to about 10, but the site seemed to continue performing fairly
    well.
  o Doing one svn checkout of fdo put one httpd up to 100% CPU for a long time.

It is my opinion that Drupal is not our problem, svn is.  Next time we are
"up against the wall" I think we should just rename /var/www/svn and see if
that doesn't bring things back under control (and verify my claim that svn is
the real pig).

Should my supposition be correct, I can see a few things we can do:
  o Get some of the junk out of svn, like those damn doxygen docs in fdo and
    mapguide's repositories.  Millions of little files is like a worst case
    I think.
  o Move the subversion and trac systems on to the secondary server.  Leave
    drupal, mailman and ldap on the primary server.

In the meantime, I think we should pull the robots.txt away or perhaps
just use it to restrict bots from some of the bulky, low value parts of
the sites, like the fdo and mapguide doxygen docs.

If the load goes up to dangerous levels, we can do the subversion test.

Best regards,
-- 
---------------------------------------+--------------------------------------
I set the clouds in motion - turn up   | Frank Warmerdam, warmerdam at pobox.com
light and sound - activate the windows | http://pobox.com/~warmerdam
and watch the world go round - Rush    | President OSGeo, http://osgeo.org