Marketing
A/B testing
Also called
- Split testing
- Experimentation
An A/B test is an experiment. Its value comes entirely from the randomization: because the two groups differ only by chance and the variant, a difference in outcome can be attributed to the variant.
The discipline is in the statistics, and it is where most tests go wrong. Stopping as soon as the result looks good ('peeking') inflates false positives dramatically. Running until significance appears, without a pre-computed sample size, does the same thing more slowly.
Most sites do not have the traffic to test small changes. At a few hundred conversions a month, only large differences are detectable in reasonable time — which means the honest options are to test big swings or to stop testing and use judgment.
A testing program that reports wins it did not earn is worse than no program, because the organization acts on it.
Commonly misunderstood
What people get wrong
The claim
“The test hit 95% significance, so we can ship it.”
What is actually true
Only if the sample size was fixed in advance. Watching a dashboard until it crosses 95% will get you there eventually on two identical variants.
Where this comes up
Services where it matters
Related terms
Next step
Working through a a/b testing decision?
Tell us the situation. We will give you the tradeoffs as we see them, including when the answer is that you do not need what you are being sold.
No pitch deck. A 30-minute conversation about what you are trying to achieve.