[Webcom] Added robots.txt

Tyler Mitchell tylermitchell at shaw.ca
Wed Jan 31 16:05:13 EST 2007


A question about spiders grabbing onto our cron job link came up on  
IRC.  Is it preferred to add a robots.txt to disallow this?

I have put one up at:
I've added the /var/www/html/robots.txt

Make sense to keep it?

Tyler


More information about the Webcom mailing list