Comment by ccgreg

How does that work for non-profit web crawls like the Common Crawl Foundation?