Comment by eugenhotaj

This is because everyone is training with synchronous sgd. all gpus need to synchronize on each gradient step so tail latency will kill you.