Rookie mistake: It's not just about the FLOPS, the interconnect matters too.
There is a reason why everyone builds super computers in a large local cluster. It's so they can have a high-bandwidth low-latency interconnect between all of the compute. The second you need to shuffle partial data results between compute nodes, you have to pay the latency hop. Interconnect quality is often a huge bottleneck, ignore it at your peril.
There is a reason why everyone builds super computers in a large local cluster. It's so they can have a high-bandwidth low-latency interconnect between all of the compute. The second you need to shuffle partial data results between compute nodes, you have to pay the latency hop. Interconnect quality is often a huge bottleneck, ignore it at your peril.