Web crawling is just like most things: 80% of the results for 20% of the work. It's always the last mile that takes the most significant cost and engineering effort.
as your index and scale grow you bump into the really difficult problems:
1. How do you handle so many DNS requests/sec without overloading upstream servers?
2. How do you discover and determine the quality of new links? It's only a matter of time until your crawler hits a treasure trove of spam domains.
3. How do you store, update, and access an index that's exponentially growing?
as your index and scale grow you bump into the really difficult problems:
1. How do you handle so many DNS requests/sec without overloading upstream servers?
2. How do you discover and determine the quality of new links? It's only a matter of time until your crawler hits a treasure trove of spam domains.
3. How do you store, update, and access an index that's exponentially growing?
Just some ideas.