Comment by TexanFeller

Yeah from your description it sounds like those problems are solved by Spark. Spark doesn't persist intermediate state to Cassandra which might make it better since its in-memory(normally, you can allow spill to disk) persistence mechanisms(RDDs, Datasets) are fast, keep data near compute, and can scale up elasticity during a run.