Comment by Havoc

Think the era of respecting stuff like robots.txt is over.

I build a web-fetch tool for my own LLMs that is essentially the same tech as web-scraping except its single requests not high volume.

You can't really get anywhere these days without disguising UA etc.