Comment by lldbg

When the data no longer fits inside a compute resource (a node, or even a rack), you are by necessity going to be distributed. Communication is a fact of life when the problem-size grows. This is true also for GPU-based computing.