Ask HN: How many GPUs do you train deep models on?

I've noticed that most of my colleagues will only use up to 4 GPUs when training deep models (tensorflow & pytorch).

Why is that?

Do you train on multiple machines?