I have a suspicion that the objective nature of most AI benchmarks is tapering over the kinds of reasoning mistakes LLMs continue to struggle with.
If you give LLMs a clearly defined spec with clear, objective acceptance criteria that the LLM can evaluate itself against, they tend to do well. But I think they struggle with more ambiguity and require a lot of handholding around judgment. That kind of disambiguating work is often baked in to the definition of a benchmark.
If you give LLMs a clearly defined spec with clear, objective acceptance criteria that the LLM can evaluate itself against, they tend to do well. But I think they struggle with more ambiguity and require a lot of handholding around judgment. That kind of disambiguating work is often baked in to the definition of a benchmark.