If you're measuring a sloppy project against a well structured one with regard to how many tests fail, that'll bias success towards the sloppy codebase, which likely has worse test coverage and less robust testing in the first place. You'd essentially need to write a single test suite that works for both projects in each pair to compare fairly.
That's not to say the study is good, but I can respect their decision because the tests passing isn't necessarily correlated with the effectiveness of the agent.
That's not to say the study is good, but I can respect their decision because the tests passing isn't necessarily correlated with the effectiveness of the agent.