I’ve been on the other side of this, defending against bots.
Basically: Well-behaved and well-intentioned scraping bots are rare. You’d get a lot of users setting update rates to 60 seconds that did a new login every time and creating as much traffic as 1000 users. Then they’d release the script for integration with something people and suddenly you have 1000 people each creating 1000 times as many login requests as a single user.
Another common problem was forgetting to implement reasonable back off for failures. A lot of newbies write scripts that immediately retry on a tight infinite loop whenever something goes wrong, sending a huge stream of requests to your server if the API changes or when it goes down. Again, multiply this by many users sharing a script and it becomes a problem.
Then of course there are people trying to make a business out of extracting your company’s data, such as putting it in some other website where they can serve ads over your content or whatever (think of all of those StackOverflow scraping websites in Google)
Basically, you can’t investigate the motivations of each individual user. You just block them all.
Basically: Well-behaved and well-intentioned scraping bots are rare. You’d get a lot of users setting update rates to 60 seconds that did a new login every time and creating as much traffic as 1000 users. Then they’d release the script for integration with something people and suddenly you have 1000 people each creating 1000 times as many login requests as a single user.
Another common problem was forgetting to implement reasonable back off for failures. A lot of newbies write scripts that immediately retry on a tight infinite loop whenever something goes wrong, sending a huge stream of requests to your server if the API changes or when it goes down. Again, multiply this by many users sharing a script and it becomes a problem.
Then of course there are people trying to make a business out of extracting your company’s data, such as putting it in some other website where they can serve ads over your content or whatever (think of all of those StackOverflow scraping websites in Google)
Basically, you can’t investigate the motivations of each individual user. You just block them all.