Comment by beybol

In server logs you can distinguish the type of LLM's traffic. OAI-SearchBot/GPTBot/ClaudeBot is just indexing but ChatGPT-User/Claude-User etc. may be valuable traffic generated by user's conversation with LLM's. So it's worth choosing which ones you block, instead of blocking all of them.