I've long argued that the biggest problem with orthodox NHST for A/B testing is that you actually don't care about 'significance of effect' as much as you do 'magnitude of effect'. Furthermore, p-values tell you nothing about the range of possible improvements (or lack thereof) you're facing. Maybe you are willing to risk potential losses for potentially huge gains, or maybe you can't afford to lose a single customer and would rather exchange time for certainty.
My favored approach I've outlined here[0]. Where the problem is basically considered one of Bayesian parameter estimation. Benefits include:
1. Output is a range of possible improvements so you can reason about risk/reward for calling a test early.
2. Allows the use of prior information to prevent very early stopping, and provide better estimates early on.
3. Every piece of the testing setup is, imho, easy to understand (ignore this benefit if you can comfortably derive Student's T-distribution from first principles)
My favored approach I've outlined here[0]. Where the problem is basically considered one of Bayesian parameter estimation. Benefits include:
1. Output is a range of possible improvements so you can reason about risk/reward for calling a test early.
2. Allows the use of prior information to prevent very early stopping, and provide better estimates early on.
3. Every piece of the testing setup is, imho, easy to understand (ignore this benefit if you can comfortably derive Student's T-distribution from first principles)
[0] https://www.countbayesie.com/blog/2015/4/25/bayesian-ab-test...