Comment by danielhanchen

We made something called Divergence-300 @32 (and later @512) which tests actual inference across 32 tokens on a held out test (Terminal Bench, DeepSWE, Math etc)

We do plan to do larger benchmark suites though!