Comment by jeroenhd

How do you control for the faked user agents scrapers use? A bad scraper will be much more common in your logs than a well-behaved browser, after all.