Ask HN: Tips for software engineering sanity with Databricks notebooks?
Folks, what are your best tips and practices for getting some software engineering sanity into developing Databricks notebooks? In particular larger data pipelines that are orchestrated from something like Azure DataFactory.
It is driving me nuts to use auto-saved versions instead of clearly defined explicitly defined commits that can be proof-read, the lack of my favourite tools from my local environment, like vim, and troubles testing things properly.
I have already found some general hints like modularizing code etc, but I wanted to hear from people in the trenches, if you found some practices or set of practices that actually made a big difference in the development experience?
It is driving me nuts to use auto-saved versions instead of clearly defined explicitly defined commits that can be proof-read, the lack of my favourite tools from my local environment, like vim, and troubles testing things properly.
I have already found some general hints like modularizing code etc, but I wanted to hear from people in the trenches, if you found some practices or set of practices that actually made a big difference in the development experience?