[SAC] [OSGeo] #139: Apache - robots

[OSGeo]

#139: Apache - robots
---------------------+-----------------------
 Reporter:  sbarnes  |       Owner:  sbarnes
     Type:  task     |      Status:  assigned
 Priority:  normal   |   Milestone:
Component:  General  |  Resolution:
 Keywords:           |
---------------------+-----------------------

Comment (by TemptorSent):

 This ticket has been cold for 10 years.
 It is not clear whether proper handling of spiderbots was ever actually
 resolved properly.

 It is unacceptable to limit the total number of connections to less than
 needed for actual web users, and it is undesirable to completely stop
 indexing of discoverable data.

 Proper use of [http://www.robotstxt.org/robotstxt.html robots.txt] and
 possibly connection/rate limiting for spiders is a more appropriate
 solution.

 The current configuration should be reviewed, corrected as necessary, and
 documented on the SAC wiki before closing this ticket.

-- 
Ticket URL: <https://trac.osgeo.org/osgeo/ticket/139#comment:5>
OSGeo <http://www.osgeo.org/>
OSGeo committee and general foundation issue tracker.