A question about spiders grabbing onto our cron job link came up on IRC. Is it preferred to add a robots.txt to disallow this? I have put one up at: I've added the /var/www/html/robots.txt Make sense to keep it? Tyler