Comment by naw103

One trap when you name AI bots specifically is that is that robots.txt groups don’t merge. Crawlers will only follow the most specific group so if GPT bot has its own group named nothing under User-agent: * applies to it anymore so you need to repeat every disallowed line under each group.