I agree with other critiques here that this A/B testing calculator does little to add to the conversation, and someone who uses it would be mislead in how to interpret results.
The procedure my team uses is:
1) hypothesize an expected conversion rate and whether to use a one-sided test (if we're testing that the state of the world remains unchanged) or a two-sided test (if we're testing that that the state of the world has changed as a result of the variation)
4) after enough visitors have come to the funnel, pass the resulting conversion numbers through ABBA http://www.thumbtack.com/labs/abba/ [1] to see confidence intervals on our results
The procedure my team uses is:
1) hypothesize an expected conversion rate and whether to use a one-sided test (if we're testing that the state of the world remains unchanged) or a two-sided test (if we're testing that that the state of the world has changed as a result of the variation)
2) run those numbers through this power/sample size calculator to determine the number of visitors we need before we can analyze the experiment: http://www.stat.ubc.ca/~rollin/stats/ssize/b2.html
3) wait for traffic
4) after enough visitors have come to the funnel, pass the resulting conversion numbers through ABBA http://www.thumbtack.com/labs/abba/ [1] to see confidence intervals on our results
For further reading, I highly recommend:
http://visualwebsiteoptimizer.com/split-testing-blog/how-to-...
[1] disclaimer: my colleague wrote ABBA