What is a holdout test?
How does a holdout test work?
Split the audience into two groups. One sees the ads. The other does not. Run both over the same period. Compare outcomes.
The gap is the lift: the difference between exposed and control. Always say lift against which control. The gap, not the exposed group's total, is what the ads added.
Does a holdout test mean switching off a whole channel?
No. It is a common worry, but a holdout does not have to work that way. It can be set by audience, or by region. Only a slice goes dark.
- By audience. Users are split at random into exposed and control groups. A conversion lift study is a platform-run version. Meta's developer documentation describes lift studies that randomly assign people to test and control groups, and says access is currently limited1.
- By region. Whole regions stay quiet while others continue. That is a geo test.
Haus, a measurement vendor, describes both routes for testing Meta: a platform lift study that randomises users, or a geo holdout that assigns regions to test and control2. It sells tests, so read it as one view.
What can go wrong?
- The test is too small to detect the effect. See minimum detectable effect.
- The control group sees the ads anyway, through another channel or region.
- The design is changed after the result arrives.
Sometimes a test is the wrong tool. Read when not to run an incrementality test. For a practical walk-through, read how to run a geo holdout test.
For ZFD: A test is one method, not the whole job. We would rather say "not worth testing yet" than run one that cannot answer the question.
Worked example
Easily confused with
Holdout test vs Geo test
A holdout built from regions rather than people.
Holdout test vs Conversion lift study
A holdout run inside an ad platform.
Holdout test vs Ghost ads
A control method that records when a control user would have seen the ad.
Buzzword alert
"We ran a holdout, so the result is proven"
Usually means: some audience did not see ads, with design and size not stated.
Ask: how was the holdout group chosen and sized?
"Always-on holdout"
Usually means: a standing control group, which is only useful if it stays clean.
Ask: what stops the holdout group from seeing our ads anyway?
FAQ
Do I have to switch off all my ads to run a holdout test?
No. You hold out part of the audience or some regions. The rest keep running as normal.
Is a holdout test the same as an A/B test?
Similar logic, different question. An A/B test compares two versions of something. A holdout compares showing ads with not showing them.
Who should sign off the design?
Someone who is not paid by the result, and before the result is known. Agree the question and the decision rule first.
Sources
- Lift Study (Meta Marketing API)Primary source
- Haus: Understanding Meta incrementality testingVendor
Let’s find what’s actually working.
Next step. See what a measured result would change in your numbers. Use the Is your ROAS real? calculator.