It is better to differentiate between the crawls, like which ones are training and which ones are search bots. You can disallow the training bots explicitly in the robots.txt and still show up in ChatGPT for example, when someone asks a question.
In my case, my logs were flooded with training crawlers, not the search bots.
In my case, my logs were flooded with training crawlers, not the search bots.