> 75% of TestGen-LLM's test cases built correctly, 57% passed reliably, and 25% increased coverage.
The problem I have with LLM generated tests is that it seems highly likely that they'd "ratify" buggy behavior, and I'd think that'd be especially likely if the code-base already had low test coverage. One of the nice things about writing new tests by hand is you've got someone who can judge if it's the system being stupid or if it's the test.
At a minimum they should be segregated in a special test folder, so they can be treated with an appropriate level of suspicion.
The problem I have with LLM generated tests is that it seems highly likely that they'd "ratify" buggy behavior, and I'd think that'd be especially likely if the code-base already had low test coverage. One of the nice things about writing new tests by hand is you've got someone who can judge if it's the system being stupid or if it's the test.
At a minimum they should be segregated in a special test folder, so they can be treated with an appropriate level of suspicion.