I find it fascinating that some of the main areas people are interested in are using benchmarks with quite few questions: ~200 for coding and ~150 for science.
As long as that's the case, it's no surprise that there is often a perceived difference of quality between benchmarks and real-world use.
I know that running the full composite benchmark set is probably already quite costly, and also that the benchmarks have to be rotated regularly, but it feels like a significant untackled problem.
As long as that's the case, it's no surprise that there is often a perceived difference of quality between benchmarks and real-world use.
I know that running the full composite benchmark set is probably already quite costly, and also that the benchmarks have to be rotated regularly, but it feels like a significant untackled problem.