Comment by Denzel

Why not just use Scrapy[1]? It's built for this sort of thing, easily extensible, and written in Python.

[1] http://scrapy.org/