Discussion How would you build an internet scale web crawler? What techniques would you use to build an internet scale web crawler/search engine.* How would you schedule and manage the crawlers?* How would you collect and parse incoming data?* What would you use to store the data?* How would you process and draw meaning from the data?* What information would be important to build a pagerank system for searching?* What other major components are missing? 42 points JustinGarrison 8 years ago 15 comments
* How would you schedule and manage the crawlers?
* How would you collect and parse incoming data?
* What would you use to store the data?
* How would you process and draw meaning from the data?
* What information would be important to build a pagerank system for searching?
* What other major components are missing?