[SAC] Motion: robots.txt, Drupal and performance

[Jason Birch]

Not sure that I get a vote, but I reluctantly have to agree with Howard.
Is there a way that we can allow the bots to at least spider the main
page, but ignore anything else?
 
Honestly, I really doubt that we'll get to 500 pages / second with
Drupal without throwing a more agressive caching strategy (Squid?) or a
better-performing CMS (is there such a thing?) in place.  Or going to
static html :)
 
But what are the options?  As Howard has pointed out, we have limited
volunteer server admin resource, so what can we do with what we've got? 
 
Lets turn off as many modules as we can because each one requires
additional SQL queries.  Drupal can be a SQL pig; I've seen reports of
100+ queries to generate a single page, which is unimaginable.
 
Is there anyone in the group with MySQL tuning experience?  If we can
get MySQL set up as effectively as possible in DW mode, this will help
Drupal as well.
 
We should consider turning off content-compression in Apache.  This
isn't nearly as expensive as MySQL, but...
 
Another thing that should be looked into is why the server is responding
in a way that invalidates caches.  This makes the bots see our content
as highly dynamic, and the'll come back for it way more frequently.
 
Jason

________________________________

From: sac-bounces at lists.osgeo.org [mailto:sac-bounces at lists.osgeo.org]
On Behalf Of Swapnil Hajare
Sent: Thursday, March 29, 2007 08:17
To: System Administration Committee Discussion/OSGeo
Subject: Re: [SAC] Motion: robots.txt, Drupal and performance


Hi
   as far as optimizing drupal performance is concerned, it looks like
some unnecessary modules installed can reduce the performance. See
http://drupal.org/node/32091 for some possible ways of encountering
performance hit. 
  As far as I remember, we have Access Stats module enabled, which is
not useful really after having installed awstats for access logs. Maybe
we should disable it and also find out more such modules which are not
used at all. 

regards,
-- 
Swapnil Hajare 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.osgeo.org/pipermail/sac/attachments/20070329/0b622750/attachment.html