Worked in the GPU/TPU validation at Google, and this tracks with something we ran into constantly: pinning down raw matrix-core throughput at the instruction level is necessary but not sufficient.
The divergence between synthetic and production numbers we kept hitting wasn't from ALU throughput — it was memory bandwidth contention once multiple kernels shared HBM, and thermal throttling on sustained runs that never shows up in short burst benchmarks. A model like this would need a sustained-load / multi-tenant term to match what we actually measured in prod.
Also relevant to varispeed's RTX5080-vs-H100 collapse above — divergence that only shows up over many epochs, not the first few, usually points to accumulated numerical or thermal drift rather than a single wrong instruction.
The divergence between synthetic and production numbers we kept hitting wasn't from ALU throughput — it was memory bandwidth contention once multiple kernels shared HBM, and thermal throttling on sustained runs that never shows up in short burst benchmarks. A model like this would need a sustained-load / multi-tenant term to match what we actually measured in prod.
Also relevant to varispeed's RTX5080-vs-H100 collapse above — divergence that only shows up over many epochs, not the first few, usually points to accumulated numerical or thermal drift rather than a single wrong instruction.