> The minimum threshold odds [to stop showing a variation] are calculated by 40% / number enabled variations
Uh... How did you choose those cutoffs? Looks like you have a significant chance of making the wrong choice.
Also:
> Once enough data is collected to start making conclusions (1000 sends per variation)
You should check out the Bayesian solution to the Multi-Armed Bandit problem. It's very close to what you are doing, but makes decisions much faster than you do because it isn't deciding to turn off a variation, merely to scale it down.
Uh... How did you choose those cutoffs? Looks like you have a significant chance of making the wrong choice.
Also:
> Once enough data is collected to start making conclusions (1000 sends per variation)
You should check out the Bayesian solution to the Multi-Armed Bandit problem. It's very close to what you are doing, but makes decisions much faster than you do because it isn't deciding to turn off a variation, merely to scale it down.