Comment by marcinzm

Our experience with Dask in distributed mode has been really painful to be honest. There's ways to shoot yourself in the foot that lead to random data corruption like changing a dataframe index with map_partitions without resetting the index afterwards. Not failure but successful runs with corrupted data. Sometimes your job gets bottlenecked on a single worker for no apparent reason so you cancel and rewrite it in a slightly different way and then it works fine. We had bugs in two of the last four versions that prevented our pipeline from running properly. If you have a dataframe of numpy arrays then unmanaged memory leaks so you need to convert them to python lists first. Again no warning by dask about this. Getting things to run efficiently requires a lot of thought about partitions, the operations you'll be doing, what indexes you have and so on.