Comment by cwbriscoe

I am not well versed in this problem but can't the web servers rate limit by known IP addresses of these crawler/scrapers?