Comment by woeirua

I understand that for some people this might help, but frankly I find all of these "reproducibility" frameworks fall flat as soon as truly big data enters the picture. Data versioning is not sufficient, because I typically cannot roll back my datasets to a previous version (and we moved forward for a reason).

Also, we are deliberately not using Databricks for this to avoid vendor lockin for something that will almost certainly be open-source soon.