Comment by metronius

If you would crawled web using Foreach(links) { get(link)} you would end up crawling useless automatically generated webpages.