My sense is that these benchmarks are not realistic in terms of the way the model is used. People building specialized AI systems are not, in my experience, letting users just chat with a base model, they would have some variant of RAG plus some guardrails plus other stuff (like routing to pre-written answers for common question).
So what use case does this test setup reflect? Is there a relevant commercial use case here?
So what use case does this test setup reflect? Is there a relevant commercial use case here?