Comment by joeswartz

Apache Beam is a quite nice and flexible tool. If you prefer Scala, then you can try Scio: https://github.com/spotify/scio.