I wish AA benchmarked the models at the reasoning efforts people actual use them at. For example, it's very rare for people that I know to use the "max" reasoning setting - specially because Claude Code itself scares you away from using it. But all of these coding benchmarks are based on just that reasoning effort.
I get it, they can only benchmark so much. But I'd rather have a benchmark of how the model is actually used and offered to customers rather than the best case scenario for it.
I get it, they can only benchmark so much. But I'd rather have a benchmark of how the model is actually used and offered to customers rather than the best case scenario for it.