I did a lot of A/B testing, but I think the examples that are used in a lot of articles about A/B testing are weird.
For small changes like change the color / appearance of a button, the difference in conversion rate is not measurable. Maybe if you can test with traffic in the range of >100K unique visitors (from the same sources), you can say with confidence which button performed better.
But how many websites / apps really have >100K uniques? If you have a long running test, just to gather enough traffic, changes are some other factors have changed as well, like the weather, weekdays / weekends, time of month, etc.
And if you have <100K uniques, does the increase in conversion pay for the amount of time you have invested in setting up the test?
In my experience, only when you test some completely different pages you'll see some significant differences.
For small changes like change the color / appearance of a button, the difference in conversion rate is not measurable. Maybe if you can test with traffic in the range of >100K unique visitors (from the same sources), you can say with confidence which button performed better.
But how many websites / apps really have >100K uniques? If you have a long running test, just to gather enough traffic, changes are some other factors have changed as well, like the weather, weekdays / weekends, time of month, etc.
And if you have <100K uniques, does the increase in conversion pay for the amount of time you have invested in setting up the test?
In my experience, only when you test some completely different pages you'll see some significant differences.