Comment by dhruvagoyal

Absolutely incredible catch and find. Looking ahead this seems like what most companies will have to use to truly avoid ai bot crawling.

It's also an insane risk imo, this basically means that I could be banned by openai/anthropic for accidentally crawling this website which might be the ultimate cyberattack - truly dystopian