Comment by gillesjacobs

In ML, you want to test general capability of a model (generalizability), because you want it to perform well on unseen tasks. In that benchmark, the literal reference is leaking through web search, the agent can see the matching real codebase online and the commits so that's test set leakage. I know no programmer that was ever paid to rewind an existing codebase to a previous commit and implement a feature/fix a bug that exists in the next commits.