fwiw, you can share a spark session between unit tests. Even persist a spark session throughout the day so your tests run against a hot session.
Straight TDD with spark is perfectly fine if you know what you're doing. I'm not saying it's easy or there's an easy guide somewhere, but it's possible.
If you're using Pyspark via the API, it's likely an incredibly important part of your process.
Straight TDD with spark is perfectly fine if you know what you're doing. I'm not saying it's easy or there's an easy guide somewhere, but it's possible.
If you're using Pyspark via the API, it's likely an incredibly important part of your process.