robots.txt

[krangor]

Hi! Is there a way to include a robots.txt file on the web archive to tell AI crawlers to go away? (Or any other instructions for crawlers users might want to set up?)

I'm aware that some crawlers ignore 'em but that's on them, not me...

[jmduke]

I am personally of the opinion that robots.txt stuff is mostly security theatre — the bad actors are gonna, well, badly act. But as you say, that's a bit overly prescriptivist. Turning this into a backlog item!

[jmduke]

Closing this in favor of buttondown/roadmap#2934.