Comment by ianeigorndua

Are these synthetic or real-world benchmarks?

Answering myself: ”Aider’s code editing benchmark asks the LLM to edit python source files to complete 133 small coding exercises from Exercism”

Not gonna start looking for a job any time soon