Comment by rxin

The "broadcast" is pretty cheap because often you already have the data in some distributed file system, or if on a single node the network bandwidth is pretty high. The problem with a lot of the deep learning workloads is that it is very compute intensive and as a result takes a long time to run. For example, it is not uncommon to take a week to train some models.