There are appropriate solutions to the multi-armed bandit problem, and a wealth of literature out there, however this is not one of those solutions.
Here's a simple thought experiment to show that this will not 'beat A/B testing every time.' Imagine you have two designs, one has a 100% conversion rate, one has a 0% conversion rate. Simple A/B testing will allow you to pick the the winning example. Whereas this solution is still picking the 0% design 10% of the time.
For some other implementations check out the following links:
Here's a simple thought experiment to show that this will not 'beat A/B testing every time.' Imagine you have two designs, one has a 100% conversion rate, one has a 0% conversion rate. Simple A/B testing will allow you to pick the the winning example. Whereas this solution is still picking the 0% design 10% of the time.
For some other implementations check out the following links:
For Dynamic Resampling:
http://jmlr.csail.mit.edu/papers/volume3/auer02a/auer02a.pdf
http://www.mynaweb.com
For Optimal Termination Time:
http://blog.custora.com/2012/05/a-bayesian-approach-to-ab-te...