Comment by kohlerm

There is no point running something like Hadoop or Spark or Flink if your problem is not bottlenecked by the resources of of a single machine. E.g. how would this solution work if the data is so large or the problem requires more context (looking at more than one line at the time) and therefore more memory? Answer: it wouldn't .