Comment by CleanCoder

Interesting read (from HN today) about crawling a quarter billion webpages in 40 hours, for $580, over 10 years ago.