ac132

Good feedback, agree completely :)

Most managers decide things without tests. Doing this well is easier said than done, but it's the reality of why some companies outperform others.

Exceptional growth doesn't happen because you're great at a/b testing. It happens when your team is good at decision making period.

Thanks for commenting on the post. :)

To take another contrasting opinion. Who cares if you know exactly why you're going. We know in a general sense why we're growing: we constantly improve the experience of our service and make customers happier.

It's quite hard to maintain aggressive year over year growth targets. If you're pulling it off it really doesn't matter if you figured out why via a/b testing or not.

Obviously, we'd prefer stronger YOY growth without knowing the exact causes than weaker YOY growth because we wasted resources on a/b testing to get precise answers to why we're succeeding.

Beyond that, I wasn't referring to trending growth via GA to see if you're making good decisions. You can trend conversion rate by channel to see if you're getting better over time. Obviously, if conversion is going up and to the right for most channels you're making good decisions.

It's unfortunate to dismiss another view point as you're frustrated or inexperienced while ignoring the actual points made.

The biggest point being #1: Resources are finite. Time spent on A/B testing cannot be spent elsewhere. How you use your time determines your growth trajectory. A/B testing is a growth tactic, but it's not always the best use of your resources.

There is absolutely no way to solve this problem. You'll always have a trade off to make when it comes to how you spend your resources and time. You shouldn't neglect that cost when deciding if a/b testing is necessary for your organization.

Btw, keep in mind, I admit we still do some a/b testing. It's just not an key component of our growth strategy.

Agree completely. :)

Disagree. The main reason we abandoned a/b testing is it's a major resource drain. Getting better at a/b testing would be another resource drain. Time is finite & how you use it determines where you end up.

The point of the article isn't to say a/b testing doesn't work or shoudn't be done. It's simply to say many companies are mis-using their resources by following the standard thinking on a/b testing.

We could spend another month figuring out how to get better at a/b testing or we could spend our time on another activity that produces a higher return on time spent. It turns out that's what we did and the return has been much better.

The post wasn't about the "futility" of a/b testing. It's about the costs of a/b testing that are rarely considered. There's plenty of "pro" a/b testing advice in the world that leads startups to mistakenly waste resources on it.

A solid a/b testing process doesn't just happen without time & effort. You need someone smart analyzing user behavior and forming hypotheses as well as a talented design team to develop concepts that test them.

We simply acknowledged the costs of doing a/b testing well and decided our resources are better spent elsewhere.

At least Google Reader still works :-)

My point was, 5 years ago no one imagined a concept like Square would exist. Just because you can't imagine a nongovernmental organization that serves the same purpose as local health inspectors doesn't mean it's impossible to have such a thing.

On the contrary, quite a few similar organizations do exist, but only when a government monopoly doesn't kill competition in that space. For example, who in the 90s would have guessed that something like TrustE would exist. It's not that far fetched to imagine that if government wasn't policing restaurants a private organization could exist to provide such a service. Yelp doesn't do that because it's not their mission and no one is going to make that their mission since the government already monopolized that job.

I'm not saying we should have 0 government either, but if we're not willing to question the necessity of things as trivial as local government health inspectors then where do we draw the line?

You did strawman. I never said capitalism should be unconstrained. I said a lot of things but nothing to that extent. Instead of responding to what I said, you created a straw man.

Just to clarify:

"A straw man is a type of argument and is an informal fallacy based on misrepresentation of an opponent's position"

You misrepresented my position. I chastised you for not responding to what I said, but that is not the same as misrepresenting your position.