The statistical methods we use now were created in the context of long-running experiments that had to be set up in advance and run in parallel. In that situation, you have to decide up-front how many subjects to test on, and the methods reflect this.
I'd like to see someone takle creating a method aimed at our situations where results steadily trickle in. There ought to be a way to come up with adaptive thresholds such that at any given time we can ask, "Do we have statistically significant results yet, or do we keep the test running?"
I'd like to see someone takle creating a method aimed at our situations where results steadily trickle in. There ought to be a way to come up with adaptive thresholds such that at any given time we can ask, "Do we have statistically significant results yet, or do we keep the test running?"