You can still ban everyone who doesn’t respect robots.txt. My experience with a crawler is that people don’t actually provide a robots.txt far more often. I can back off on 429 but I’d rather use Crawl-Delay etc.
Bots have limited ability to announce themselves. I just use fixed IP range and UA. But the Chinese trawlerbots use smart TV residential proxies and will copy someone else’s UA.
Bots have limited ability to announce themselves. I just use fixed IP range and UA. But the Chinese trawlerbots use smart TV residential proxies and will copy someone else’s UA.