Comment by dshah

I'm going to vote for Nutch too. Have heard good things.

Also, if someone on this list wants to work on a cool web spidering project (probably using Nutch), send me a message. I'm looking for someone.