Pretty animation, but I have some reservations about the maths:
- Normal approximation, as already noticed, ain't no good. Use the Wilson score instead.
- No power calculation? Type II errors are far more important IMHO in typical web applications because switching costs are small.
- non-overlapping 95% confidence interval do not imply p < 0.05. It's actually much lower than that. 83% CI is more like p of 0.05. (Errors add in quadrature.)
There is a tension between making something simple for the lay person and providing knobs for the expert to twiddle. I can see the case for removing the knobs but the choices should at least be documented.
[It's late here so this post is a bit slim on details. If you're interested sign up to http://bandits.mynaweb.com/ as the next section covers confidence intervals.]
- Normal approximation, as already noticed, ain't no good. Use the Wilson score instead.
- No power calculation? Type II errors are far more important IMHO in typical web applications because switching costs are small.
- non-overlapping 95% confidence interval do not imply p < 0.05. It's actually much lower than that. 83% CI is more like p of 0.05. (Errors add in quadrature.)
There is a tension between making something simple for the lay person and providing knobs for the expert to twiddle. I can see the case for removing the knobs but the choices should at least be documented.
[It's late here so this post is a bit slim on details. If you're interested sign up to http://bandits.mynaweb.com/ as the next section covers confidence intervals.]