Comment by mendigou

There are multiple ways to train in parallel, and that's one of them:

https://pytorch.org/tutorials//distributed/home.html