Comment by sytelus

Common Crawl is more interesting for research purposes than for real search engine business. It refreshes only 4 times a year and has only 2 billion pages. For anything under ~10B pages is most likely to be called "research" or "toy" index. You have almost zero chance of building competitive index if you are below ~15B. Google is rumored to crawl about 150B pages as reference. There are estimated 1.2T pages out there which does not get crawled due to various reasons.