I have yet to see LLMs do well “autonomously” on anything that doesn’t fall into any of these 3 buckets:
1. well specified
2. well tested (which is a form of spec)
3. represented in the training set (i.e. not built on a corpus of human developed examples)
For example, a lot of the compiler porting efforts fall into all 3 buckets: compilers are well specified (there is a set of valid inputs and an expected behavior), they are typically well tested, and as a bonus, in a way that is implementation agnostic thanks to the clean interface (think: integration vs unit tests), and are very well represented in the training set.
1. well specified
2. well tested (which is a form of spec)
3. represented in the training set (i.e. not built on a corpus of human developed examples)
For example, a lot of the compiler porting efforts fall into all 3 buckets: compilers are well specified (there is a set of valid inputs and an expected behavior), they are typically well tested, and as a bonus, in a way that is implementation agnostic thanks to the clean interface (think: integration vs unit tests), and are very well represented in the training set.