I’ve run almost exactly the same number of experiments as the author, and experienced the same frustrations until recently.
It sounds as though the author needs to combine the "How we improve conversion without A/B testing" section with user tests and data analysis to build stronger hypotheses. Since we've put more effort into the research phase, our success rate has improved dramatically.
The real value of A/B testing is validating meaningful hypotheses that help you learn what matters to your customers, not unrelated individual improvements. By only observing the overall trend, you miss out on this.
It sounds as though the author needs to combine the "How we improve conversion without A/B testing" section with user tests and data analysis to build stronger hypotheses. Since we've put more effort into the research phase, our success rate has improved dramatically.
The real value of A/B testing is validating meaningful hypotheses that help you learn what matters to your customers, not unrelated individual improvements. By only observing the overall trend, you miss out on this.
I wrote about this recently here: https://medium.com/@nickboyce/5-steps-to-a-better-a-b-testin...