Comment by gok

It's interesting how much power we end up spending purely on DRAM transfer. HBM3 uses about 5 pJ/bit, so maxing out an H100's 3.35TB/s of bandwidth requires around 135 watts just for the RAM. Given how many AI workloads are entirely bandwidth-bound, you could probably build a much cheaper chip that just went all-in bandwidth and got rid of all the fancy compute elements.