Comment by taneq

I can't find it now but there was a great blog post about parallel processing of large data sets just using Linux streams from the command line. They ended up comparing a single laptop (iirc) against a Hadoop cluster and coming out ahead.