What is an A/B test?
How does an A/B test work?
People are split at random, so the two groups are alike. Each group sees one version. You change one thing, choose the metric before you start, and wait until there is enough data to judge.
Decide two things before the test. First, what decision will it change? Second, what result would make you stop? Without both, a test is a hobby.
A/B test or incrementality test?
They answer different questions. An A/B test compares two versions of something. A holdout test compares ads against no ads, to see what the ads added. That second question is incrementality: what would have happened without the activity.
So an A/B test can tell you which ad is better. It cannot tell you whether running ads at all is worth the money. Lift is the term for the difference between the exposed group and the control group.
A paper on designing incrementality experiments on Facebook notes that lift studies differ from standard A/B tests, and gives its own power and sample size calculation for them1. So do not copy A/B sizing rules onto an ad lift test.
For ZFD: We use A/B tests for creative and page choices, and a holdout or geo test when the question is whether the ads work. A test is one method, not the whole job.
What goes wrong with A/B tests?
- Too many tests for the traffic. Small audiences produce noisy answers. How many tests make sense is in how many A/B tests make sense at your traffic.
- Stopping when the result looks good. Choose the sample size first, then wait for it.
- Judging on the wrong metric. Clicks went up. Did margin?
Worked example
Easily confused with
A/B test vs Holdout test
Ads versus no ads, not version versus version.
A/B test vs Statistical significance
A check on whether a difference is unlikely to be chance. It does not say the difference is big enough to matter.
A/B test vs Lift
The measured difference between exposed and control groups, as incremental conversions or a percentage.
FAQ
How long should an A/B test run?
Until it reaches the sample size you chose before starting. That depends on your traffic and on the size of the effect you need to see.
Is an A/B test the same as an incrementality test?
No. An A/B test compares two versions. An incrementality test compares having ads with not having them.
Can a small shop run A/B tests?
Yes, but fewer, with bigger changes between versions. Small differences need more data than a small shop often has.
Let’s find what’s actually working.
Next step. Check what your numbers look like on explicit assumptions. Use the Is your ROAS real? calculator.