I guess I'm not very versed in website A/B testing, but wouldn't it be much better to analyze these results in a regression framework where you can correct for the covariates?
On top of this, logistic regression makes your units a lot more interpretable than just looking at differences in means. I.E. The odds of buying something are 1.1 when you are assigned in group B.
On top of this, logistic regression makes your units a lot more interpretable than just looking at differences in means. I.E. The odds of buying something are 1.1 when you are assigned in group B.