Comment by mushfiq

Clean and direct write up. It reminds me when I used to work on crawling/scrapping. Covered most of the topics that needs to know for web crawling.