Comment by geekrax

I have replaced all robots.txt rules with simple WAF rules, which are cheaper to maintain than dealing with offending bots.