Comment by rpearcea

Interesting, but 1.3 million pages is somewhat limited. They seem to have done a good job indexing Wikipedia. I'm curious, why not scan the full ipv4 address space and index the main page of every website you find?