Author here. I’ve been running this experiment for a couple of years whenever a new model comes out, and this is the first time the results have been good enough to share.
I provide an LLM with a natural language specification (e.g., "an HTTP server on port 8125"), and a bash loop executes what the model generates; the actual error encountered is then fed back to the model until the test passes. There are three levels: C with gcc, assembly with as + ld, and raw bytes (where the model writes the ELF file directly, with nothing between it and the binary other than `xxd -r`).
The raw bytes level isn't always reliable: in 5 runs using Claude Sonnet 5.5 on x86-64, "Hello, World" succeeded in 4 out of 5 attempts, and the HTTP server in 2 out of 5. The C and assembly levels succeeded in all 5 attempts. The system also executes binaries generated by the model, so please check what the code does before running it.
I provide an LLM with a natural language specification (e.g., "an HTTP server on port 8125"), and a bash loop executes what the model generates; the actual error encountered is then fed back to the model until the test passes. There are three levels: C with gcc, assembly with as + ld, and raw bytes (where the model writes the ELF file directly, with nothing between it and the binary other than `xxd -r`).
The raw bytes level isn't always reliable: in 5 runs using Claude Sonnet 5.5 on x86-64, "Hello, World" succeeded in 4 out of 5 attempts, and the HTTP server in 2 out of 5. The C and assembly levels succeeded in all 5 attempts. The system also executes binaries generated by the model, so please check what the code does before running it.
I’d be happy to answer any questions