zephraph
Joined 10 years ago
Joined 10 years ago
Steve, if you come to NYC hit me up!
I feel it's significantly more practical than something like nix which feels like it has a steep learning curve.
This doesn't actually work out that well in practice though because the implementations the llm tended to generate were highly specific to pass the tests. There were several times it would cheat and just return hard coded strings that matched the expects of the tests. I'm sure better prompt engineering could help, but it was a fairly funny outcome.
Something I've found more valuable is generating the tests themselves. Obviously you don't wholesale rely on what's generated. Tests can have a certain activation energy just to figure out how to set up correctly (especially if you're in a new project). Having an LLM take a first pass at it and then ensuring it's well structured and testing important codepaths instead of implementation details makes it a lot faster to write tests.
I generally agree that the workflow could be improved though.
Reach out to me at [email protected]