Comment Comment by jakemanger I've been building whatmodel, an open source AI leaderboard, and wanted to see how a labs' own benchmark scores compare with independent testers.Using 1,629 published benchmark results, I matched 611 comparisons of 180 models on 103 benchmarks.AI labs reported scores 6.4% higher on average for the same models on the same tests.The dataset is CC BY-SA 4.0 and the code is MIT. Sources and their licenses are recorded, with links on each model's page.Data and code: https://github.com/jakemanger/whatmodelPlease star on Github if you find it useful. Contributions are very much welcome! parent jakemanger 13 days ago
Using 1,629 published benchmark results, I matched 611 comparisons of 180 models on 103 benchmarks.
AI labs reported scores 6.4% higher on average for the same models on the same tests.
The dataset is CC BY-SA 4.0 and the code is MIT. Sources and their licenses are recorded, with links on each model's page.
Data and code: https://github.com/jakemanger/whatmodel
Please star on Github if you find it useful. Contributions are very much welcome!