[SAC] SWAP space filled

[Frank Warmerdam]

sbarnes wrote:
> I just did a quick check of the logs and it may have been a few things.
> 
> the httpd server was restart around 15:54 yesterday by Tyler around that 
> time
> 
> trac was doing a changeset in ossim
> multiple robots where spidering, msnbot, slurp, twiciler, and teoma
> 
> the teoma (ask jeeves) robot was crawling through our mail archives
> 
> 65.214.45.100 - - [04/Feb/2008:15:53:48 -0500] "GET 
> /pipermail/grass-commit/2001-May/000154.html HTTP/1.0" 404 263 "-" 
> "Mozilla/5.0 (compatible; Ask Jeeves/Teoma; 
> +http://about.ask.com/en/docs/about/webmasters.shtml)
> 
> i suspect it was probably the combo, that pushed our load up.

Shawn,

Can you find any additional particulars on the ossim changeset request?

I stress that in the past the problem was *entirely* caused by huge
changeset requests from some spiders.  One big changeset request causes
massive swapping as it is assembled in RAM and then the load average
spikes from normal other activity because nothing is getting completed
due to io contention.

We have a "spider trap" laid to try and identify spiders that are roaming
through trac changesets despite the robots.txt request not too.  If the
spider hasn't fallen into the trap yet, but we can identify IP#'s we can
just add them to /var/www/trac/forbidden_ips.txt to block them.

I don't think we need to sweat about spiders in legal areas like the mailing
list archives.

Best regards,
-- 
---------------------------------------+--------------------------------------
I set the clouds in motion - turn up   | Frank Warmerdam, warmerdam at pobox.com
light and sound - activate the windows | http://pobox.com/~warmerdam
and watch the world go round - Rush    | President OSGeo, http://osgeo.org