Yes. There's also other (better) ways to solve this issue: for instance sampling input/outputs in production and srtting them in stone in the tests.
An issue with going with llms will be to validate if the behavior described are merely tolerated or if they're correct. Another will be wether something is actually tested (e.g. a code change still wouldn't break the test). Too granular output check would be an issue as well.
All in all this feels like a bad idea, but I hope to be wrong.
An issue with going with llms will be to validate if the behavior described are merely tolerated or if they're correct. Another will be wether something is actually tested (e.g. a code change still wouldn't break the test). Too granular output check would be an issue as well.
All in all this feels like a bad idea, but I hope to be wrong.