Author here. I ran 40 ports of my codebase (Python to TypeScript and back). I accidentally left pycache in the sandbox, which the agents used to decompile the bytecode and reverse-engineer the tests I'd withheld - so half the results went in the bin, along with anything about the effect of tests on port quality.
I learned cheating is hard to stop. I didn't answer whether the ports are faithful - more on that in the article.
I learned cheating is hard to stop. I didn't answer whether the ports are faithful - more on that in the article.
Also have reproducible research repo here: https://github.com/thekevinscott/ports-research