[SAC] Motion: robots.txt, Drupal and performance

[Tyler Mitchell]

On 29-Mar-07, at 8:45 AM, Jason Birch wrote:

> Not sure that I get a vote, but I reluctantly have to agree with  
> Howard.  Is there a way that we can allow the bots to at least  
> spider the main page, but ignore anything else?

Isn't there a way to feed search engines our site map directly?

> But what are the options?  As Howard has pointed out, we have  
> limited volunteer server admin resource, so what can we do with  
> what we've got?

I'm also trying to find more people who are familiar with tuning  
drupal.  (Swapnil keep the ideas coming and put into the wiki!)  Ejer  
and alienbrain (on IRC) may be able to help too.

> Another thing that should be looked into is why the server is  
> responding in a way that invalidates caches.  This makes the bots  
> see our content as highly dynamic, and the'll come back for it way  
> more frequently.

Cache was currently set to 5min lifetime, perhaps it should be more now.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.osgeo.org/pipermail/sac/attachments/20070329/e10e7aaf/attachment.html