What is sample size in marketing tests?
Why does sample size matter?
A test compares two groups. If each group is tiny, a few lucky orders can swing the result. You see a gap. It may be chance.
A bigger sample smooths this out. The more conversions you observe, the less one odd day or one big order can move the answer.
With ads, "bigger" can mean very big. A chapter by Lewis, Rao and Reiley, drawing on large online ad experiments at Yahoo!, argues that measuring the sales effect of advertising needs very large samples, and that the intervals around the result can stay wide even then1.
What counts is usually conversions, not visitors. A test with a million views and a handful of purchases has little to say about purchases.
What decides how big a test must be?
Three things:
- How often people convert today. Rare outcomes need more observations to see.
- How small an effect you want to detect. Spotting a tiny effect takes far more data than spotting a large one. The smallest effect a design can reliably find is the minimum detectable effect.
- How sure you want to be. This links to statistical significance and to statistical power, the chance of finding a real effect when there is one.
So there is no universal threshold. The right size depends on your own volumes. A brand with few daily orders may need weeks, or may be unable to run a clean test at all.
Does the usual A/B formula work for ad tests?
Not without checking. A paper by Liu, Bettaney and Chamberlain notes that Facebook lift studies differ from standard A/B tests, so the usual experimentation literature does not say how to work out power and minimum sample size for them2. The paper sets out those calculations for Facebook lift studies, including studies with several test cells2. Our reading: a calculator built for two web page variants is not a safe shortcut for an ad holdout test.
An A/B test on a web page and a lift test on ads are close cousins. Their sample size rules are not identical.
Worked example
Easily confused with
Sample size vs Minimum detectable effect
The size of the effect the test can see. Sample size is the amount of data needed to see it.
Sample size vs Statistical significance
A check on whether a gap is likely to be chance. A small sample makes it hard to pass that check, even when the effect is real.
For ZFD: If a test cannot be big enough at your volumes, we would say so before it starts. An underpowered test only produces an answer that cannot be used.
FAQ
Is a bigger sample always better?
Bigger gives more certainty, but it costs time and money. The goal is a sample big enough to answer the question you set, no more.
What if my test is too small?
A small test can still be worth running if you treat the result as weak evidence. Do not treat a gap you cannot separate from chance as proof either way.
Let’s find what’s actually working.
Next step. Check what your ROAS looks like on explicit assumptions. Use the Is your ROAS real? calculator.