Comment by xiaodai

Spark is a cancer. Sooner or later, 99.9% of the people using Spark will wake up to the fact that "hey, I got 1TB of RAM， why do I need this?"

Spark and PySpark are just PITA to the max.