> My controversial opinion is that most software tests should be integration tests that use randomized inputs. I doubt I'll get very many upvotes for that idea.
Probably not, but largely because you used the word "randomized". One of the most annoying things to deal with in a CI workflow is flaky tests (where something unchanged will randomly fail, or re-running will result in a different outcome).
The most common way I've seen this happen is with an external dependency like using a database server or calling a 3rd party API. The failure is usually temporary (eg: timeout) and so it's a false negative. This trains developers to do a lazy workaround: re-run the build until it passes.
True "randomized" data has the possibility that you get all kinds of flaky tests, but that they cause false positives: the failures are legitimate, passing is not.
Pseudo-randomized [1] data is a different story (see: fuzzing [2]), because you still have predictable tests -- the difference is a human doesn't have to think up thousands of test cases. I'd say it's also important to add specific test cases when you do find a specific type of failure, just to be sure that always gets tested.
I think if you call it "pseudo-random" or "fuzz testing" you'll find it's much less controversial.
[1] https://en.wikipedia.org/wiki/Pseudorandomness
[2] https://en.wikipedia.org/wiki/Fuzzing
Probably not, but largely because you used the word "randomized". One of the most annoying things to deal with in a CI workflow is flaky tests (where something unchanged will randomly fail, or re-running will result in a different outcome).
The most common way I've seen this happen is with an external dependency like using a database server or calling a 3rd party API. The failure is usually temporary (eg: timeout) and so it's a false negative. This trains developers to do a lazy workaround: re-run the build until it passes.
True "randomized" data has the possibility that you get all kinds of flaky tests, but that they cause false positives: the failures are legitimate, passing is not.
Pseudo-randomized [1] data is a different story (see: fuzzing [2]), because you still have predictable tests -- the difference is a human doesn't have to think up thousands of test cases. I'd say it's also important to add specific test cases when you do find a specific type of failure, just to be sure that always gets tested.
I think if you call it "pseudo-random" or "fuzz testing" you'll find it's much less controversial.
[1] https://en.wikipedia.org/wiki/Pseudorandomness
[2] https://en.wikipedia.org/wiki/Fuzzing