Comment by murtza

Have you taken a look at the Scrapy framework for Python?

http://scrapy.org/