lusob

Author here. I’ve been running this experiment for a couple of years whenever a new model comes out, and this is the first time the results have been good enough to share.

I provide an LLM with a natural language specification (e.g., "an HTTP server on port 8125"), and a bash loop executes what the model generates; the actual error encountered is then fed back to the model until the test passes. There are three levels: C with gcc, assembly with as + ld, and raw bytes (where the model writes the ELF file directly, with nothing between it and the binary other than `xxd -r`).

The raw bytes level isn't always reliable: in 5 runs using Claude Sonnet 5.5 on x86-64, "Hello, World" succeeded in 4 out of 5 attempts, and the HTTP server in 2 out of 5. The C and assembly levels succeeded in all 5 attempts. The system also executes binaries generated by the model, so please check what the code does before running it.

I’d be happy to answer any questions

This is a Python program that uses OpenAI's ChatGPT to generate code review comments from diffs. The program reads the diff changes from the standard input, which allows piping from git diffs (or any other diff tool) directly into the program.

git diff master..branch | python3 chatgpt_code_reviewer.py