PashaGo
Joined 5 months ago
Karma
9
Submissions
14
Member since
Jun 2
Joined 5 months ago
Also, model providers are not interested to have their models compared head-to-head under identical conditions. And “Model A is better than Model B” is almost meaningless by itself. Better for what task? With what prompt? What inputs? What budget? What failure tolerance?
It would be nice to have a place where users could run their own benchmarks, define evaluation criteria for their actual use cases, and make those runs verifiable by others.
In the end, it became heavy overengineering: people no longer understood not only the repo itself, but also the extra layer describing it. Meanwhile, coding assistants are already quite good at reading codebases directly.