wiredfool
Joined 17 years ago
Karma
6,556
Submissions
2,462
Member since
Jul 6
[email protected]
View on Hacker NewsJoined 17 years ago
Daphne (The Cat) Lastname.
* Sometimes there's a pattern to the country. Oftentimes, not.
* User-agent, rotated between common, valid, current web browsers.
* Other headers, sec-*, accept, etc, generally valid and rotating.
* Bots will load the site to saturation in a denial of wallet attack.
The only thing that's specific is: * urls have a pattern.
* it's obviously invalid traffic.
(non-bot traffic on my sites does not go from 0 to 200r/sec on the search interface in seconds. It does not go away that fast either)The crawlers get stuck in a loop requesting the dataset listing page with every. single. combination. of. facets. At essentially as fast as it can be pumped out or blocked.
Had one bot super interested in a single organization to the tune of 1000 r/min for 24+ hours. Realistic rotating user agents, realistic sec-* headers, no ip address seen more than a couple times in 10 minutes. The only commonality was the route.