Comment by knicholes

This seems pretty simple to me to do. Search the html of the main page for anchor tags. Add the links in those tags to an array as your exploration frontier. Once done parsing that html, load the next link. Add deduplication to avoid loops and just run a depth-first search. What am I missing?