Comment by eugenhotaj

The network communication overhead would be way too high to make this useful. At least for current methods of training large models.