Comment by felixfbecker

OpenAI and Anthropic respect robots.txt afaik