I generally agree, but there's one thing that bothers me about the practice:
Before disabling the test, does anyone even consider the possibility that maybe it's not the test that's flaky, but that the software under test is itself buggy in a non-deterministic way?
I don't have a good answer for how to deal with this. It would be an enormous amount of effort to go through every flaky result and reason about whether the problem is with the test or the code under test, especially in larger codebases. Maybe the heuristic of assuming that it's the test's fault is good enough, and this is what I've done in every team I've worked in.
Before disabling the test, does anyone even consider the possibility that maybe it's not the test that's flaky, but that the software under test is itself buggy in a non-deterministic way?
I don't have a good answer for how to deal with this. It would be an enormous amount of effort to go through every flaky result and reason about whether the problem is with the test or the code under test, especially in larger codebases. Maybe the heuristic of assuming that it's the test's fault is good enough, and this is what I've done in every team I've worked in.
But it does bother me.