"I started experimenting with Nvidia's RTX 4090s. I bought a bunch of them and put them into a mining rack and just ran some tests. I quickly figured out that is not the way to go,"
Well, I hope they were at least smart enough to use PCEI4 x16...otherwise that will never, ever work well. 2x 4090 saturate my PCEI4 x16 bus during training workloads. If they were PCEI3 x1 risers... That's 32x lower inter-gpu bandwidth!
Also, for 50K per GPU, and all this hacking, buy an older 8x A100 system with water-cooling or something.
Well, I hope they were at least smart enough to use PCEI4 x16...otherwise that will never, ever work well. 2x 4090 saturate my PCEI4 x16 bus during training workloads. If they were PCEI3 x1 risers... That's 32x lower inter-gpu bandwidth!
Also, for 50K per GPU, and all this hacking, buy an older 8x A100 system with water-cooling or something.