[SAC] Added robots.txt

[Jason Birch]

Perhaps an "allow from" entry for that file would be better, set to only
allow from the machine that triggers the cron job?

robots.txt is only advisory

Jason 

-----Original Message-----
From: sac-bounces at lists.osgeo.org [mailto:sac-bounces at lists.osgeo.org]
On Behalf Of Tyler Mitchell
Sent: Wednesday, January 31, 2007 13:05
To: System Administration Committee Discussion/OSGeo; Web committee
discussions
Subject: [SAC] Added robots.txt

A question about spiders grabbing onto our cron job link came up on IRC.
Is it preferred to add a robots.txt to disallow this?

I have put one up at:
I've added the /var/www/html/robots.txt

Make sense to keep it?

Tyler
_______________________________________________
Sac mailing list
Sac at lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/sac