Exactly. We'd get customers thats sites would drown in bot traffic because they pulled the same shit, changing user agents, different ips, etc. I had to build custom mod security rules to block the patterns these boys would pull. What's funny is that the bots would have a site where you could control the crawl rates but it's just a placebo. They would crawl even if you requested them to stop.
The issue with bots hosted on AWS or any cloud for that matter is that as a web host you can't just block the IPs because legitimate traffic comes from them in the form of CMS plugins, backups, etc.
The issue with bots hosted on AWS or any cloud for that matter is that as a web host you can't just block the IPs because legitimate traffic comes from them in the form of CMS plugins, backups, etc.