> It’s also a “you can’t scrape us but we can scrape you” mentality these companies are getting into.
And the problem often isn't some advanced attack. It's enough to follow links with session params, generate endless urls or ignore robots.txt to turn a regular crawler into a load generator. A site owner needs crawler protection from specific behavior and not another list of known bot signatures
And the problem often isn't some advanced attack. It's enough to follow links with session params, generate endless urls or ignore robots.txt to turn a regular crawler into a load generator. A site owner needs crawler protection from specific behavior and not another list of known bot signatures