So actually I've been testing this — it seems to be useful in limited cases. LLMs have gotten so good now that a properly defined spec is no worse than a spec + property based testing.
I still found it useful for
1. trusting a bit of code is correct
2. rule-dense bits of code (eg something like a date parser)
The real issue so far is that I'm having trouble finding tasks where an LLM can't correctly code up an exhaustively defined spec.
I still found it useful for
1. trusting a bit of code is correct
2. rule-dense bits of code (eg something like a date parser)
The real issue so far is that I'm having trouble finding tasks where an LLM can't correctly code up an exhaustively defined spec.