Comment by CoastalCoder

This seemed like a big topic when I was interviewing with Meta and nVidia some months ago.

Meta had a few good YouTube videos about the problems of dealing with this many GPUs at scale.