[PROJ] robots.txt

[Javier Jimenez Shaw]

Hi

In proj.org/robots.txt there is currently this
User-agent: *

Disallow: # Allow everything

Sitemap: https://proj.org/sitemap.xml

while in gdal.org/robots.txt this
User-agent: *
Allow: /en/latest/
Disallow: /en/

That means that any robot is indexing aaaall the proj pages, including old
versions (I checked it in Google Console).

Should we do the same in PROJ as in GDAL?

Cheers,
Javier.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/proj/attachments/20240916/1362e5b2/attachment.htm>