I could not disagree with this more. I remember vividly having this "low-intent" vs "high-intent" debate at Thumbtack, when we rolled out changes that A/B tests showed increased conversion (by a lot), but some people in the company thought the changes were ugly and "off-brand" and argued they brought in the wrong type of customers. So we ran the test again that we knew raised conversion by a lot, and then followed the 2 cohorts of customers and watched their behavior. The control group vs the 10% more from whatever the test was that increased conversion. They behaved exactly the same. They came back again at the same rates. They made the same amount of profit (per customer). Their response rates to emails were the same. They closed jobs at the same rates. As far as we could tell they were identical.
I have to admit I was a little surprised too, but for our business it didn't seem this "high-intent" vs "low-intent" distinction existed. And with that out of the way we continued to optimize conversion rates, and our revenue continued to go up.
Every company is different so I don't want to generalize too much, but if somebody tells me they ran an A/B test that said some key flow went up 10%, but then afterwards the traffic/revenue/whatever didn't go up 10%, I think the most likely candidate is bad test design. Humans are really good at rigging A/B tests to produce wrong results in their favor. I guarantee every single company who isn't maniacal about A/B testing does at least one of the following:
- Uses a tool to grade A/B tests that isn't statistically sound
- Let's people check tests too often and allows them to stop the test when it hits a good result
- Running a test with a lot of similar variations and cherry picking the best one
- Doesn't plan for enough traffic to detect the percentage of change their test is likely to produce
All of these create the potential for the perceived gains of the A/B test not matching up with real world result.
I'm not saying the distinction between "low-intent" and "high-intent" customers doesn't exist, but it is fairly easy to test for. Do that test for your business and see if that distinction exists. But don't use it as some magical explanation for why your A/B tests aren't producing the results you want as this article suggests.
I have to admit I was a little surprised too, but for our business it didn't seem this "high-intent" vs "low-intent" distinction existed. And with that out of the way we continued to optimize conversion rates, and our revenue continued to go up.
Every company is different so I don't want to generalize too much, but if somebody tells me they ran an A/B test that said some key flow went up 10%, but then afterwards the traffic/revenue/whatever didn't go up 10%, I think the most likely candidate is bad test design. Humans are really good at rigging A/B tests to produce wrong results in their favor. I guarantee every single company who isn't maniacal about A/B testing does at least one of the following:
- Uses a tool to grade A/B tests that isn't statistically sound
- Let's people check tests too often and allows them to stop the test when it hits a good result
- Running a test with a lot of similar variations and cherry picking the best one
- Doesn't plan for enough traffic to detect the percentage of change their test is likely to produce
All of these create the potential for the perceived gains of the A/B test not matching up with real world result.
I'm not saying the distinction between "low-intent" and "high-intent" customers doesn't exist, but it is fairly easy to test for. Do that test for your business and see if that distinction exists. But don't use it as some magical explanation for why your A/B tests aren't producing the results you want as this article suggests.