Comment by krupan

I'm confused. Who made this benchmark and why? It is fascinating to me that people are putting this much work into trying to measure and evaluate LLMs when we've never gone to this level for humans. Sounds like they also built a harness/feedback loop so the LLM knows if it's doing a good job? Again, why didn't anyone ever do this for human engineers?