Comment by bbayer

I am using Scrapy a lot. http://scrapy.org/ It is very well designed web crawling library.