Cool. Sounds like a you problem, you meaning crawlers and AI trainers. Now it will fall on every web developer to tag their data for it to be exploited by megacorps?
Now that I think of it- why do we put up with robots.txt at all?
> A robots.txt file tells search engine crawlers which URLs the crawler can access on your site. This is used mainly to avoid overloading your site with requests
If someone overloads your site with automated requests how is that not criminal? Why aren't they liable?
Now that I think of it- why do we put up with robots.txt at all?
> A robots.txt file tells search engine crawlers which URLs the crawler can access on your site. This is used mainly to avoid overloading your site with requests
If someone overloads your site with automated requests how is that not criminal? Why aren't they liable?