I use S3 and EC2 interchangably when it comes to EMR, which is what I presume is what the author is referring to. Most EMR jobs consume and write their data to S3 and use a temporary HDFS cluster for scratch. By and large scratch data ends up being much, much larger than the original inputs, if for no other reason than that needed during the shuffle/sort stage. (I am assuming we are talking about non trivial map reduce jobs here, not word counters, where you have many reduce steps.) it goes without saying there are many applications where user-created functions will generate more data than they consume (combinatorics, etc)
Data locality is but one reason to use map reduce. In practice EMR allows you to draw upon elastic computing resources to allow you to process data however you like. It provides developer and cluster isolation and linearly scalable I/O from S3 as well. The author sounds like someone who may have read the academic papers and a few books but hasn't used these tools in practice.
I use S3 and EC2 interchangably when it comes to EMR, which is what I presume is what the author is referring to. Most EMR jobs consume and write their data to S3 and use a temporary HDFS cluster for scratch. By and large scratch data ends up being much, much larger than the original inputs, if for no other reason than that needed during the shuffle/sort stage. (I am assuming we are talking about non trivial map reduce jobs here, not word counters, where you have many reduce steps.) it goes without saying there are many applications where user-created functions will generate more data than they consume (combinatorics, etc)
Data locality is but one reason to use map reduce. In practice EMR allows you to draw upon elastic computing resources to allow you to process data however you like. It provides developer and cluster isolation and linearly scalable I/O from S3 as well. The author sounds like someone who may have read the academic papers and a few books but hasn't used these tools in practice.