The way benchmarks work is very very different from how you work with an agent.
The benchmarks often have ambiguous instructions, and there is no interaction with the agent. So a docker image is started, the agent is called in non-interactive mode with a prompt saying to read the instructions and finish the task. The instructions are often somewhat ambiguous and the agent has no way to ask for clarification. If it picks the wrong option out of many seemingly equally likely options that test will be mark as failed.
The benchmarks often have ambiguous instructions, and there is no interaction with the agent. So a docker image is started, the agent is called in non-interactive mode with a prompt saying to read the instructions and finish the task. The instructions are often somewhat ambiguous and the agent has no way to ask for clarification. If it picks the wrong option out of many seemingly equally likely options that test will be mark as failed.