Comment by asdkjlfkasjdf

exactly, but nobody in their sane minds would be using robots.txt to block crawlers as that is SEO (organic traffic from searches) disaster.

People could be adding a specific robot user-agent, if they knew openai even existed before yesterday and was stealing their content. but nobody did.