Comment by aonic

Check out Nutch, not sure if its exactly what you want though. It's in Java not Python, but it works with Hadoop quite nicely