OpenAI scraped aggressively
for years. Why should others put themselves behind an artificial moat?
If you want to block access to a site, stop relying on arbitrary opt-in voluntary things like user agent or robots.txt. Make your site authenticated only, that’s literally the only answer here.
If you want to block access to a site, stop relying on arbitrary opt-in voluntary things like user agent or robots.txt. Make your site authenticated only, that’s literally the only answer here.