When not to run an incrementality test (and what to do instead)

Start here. Before you design anything, write down the decision the test feeds. Then write down the results you think are plausible.

• If the test shows the channel is highly incremental, what do you do?
• If it shows the channel is barely incremental, what do you do?

If both answers are "the same as now", do not test. You would be paying to confirm a choice you have already made. If the answers differ, you have a decision threshold: a result that flips the budget. A test is worth running only when a plausible result lands on the other side of that threshold.

Also check who is asking. If the budget is fixed by contract, a launch plan or a board promise, the test cannot move it. Nobody enjoys finding that out after the invoice.

A test compares a holdout test group, which does not see the ads, with an exposed group. The question is whether the gap is real or noise.

That depends on the sample size: how many users, conversions or regions the test has. It also depends on the minimum detectable effect (MDE), the smallest lift your design can reliably pick up. In plain English: the smaller the effect you want to see, the more data you need. With low volume, your test can only detect large effects. If the true effect is moderate, you probably will not see it.

Result: you pay for the holdout and read "can't tell". A "can't tell" is a proper outcome of an underpowered test, not a failure of the team. Economists who ran large advertising experiments at Yahoo! found that ad effects are small next to the normal swings in what people buy. They report that even cleanly designed experiments typically needed over a million people to measure an economically meaningful effect, and that some were still underpowered¹. Most brands work with far smaller numbers.

Statistical significance will not rescue you either. It tells you whether a difference is unlikely to be chance. It does not say whether the difference is big enough to matter for your budget.

Published entry thresholds for platform-run studies vary by source. A trade-media report says Google lowered the minimum spend for its incrementality testing². The same publication's overview of conversion lift quotes separate minimums for Google's studies, citing Google's help pages without linking them³. For a Meta conversion lift study, two vendor pages set the bar differently: one as a total number of conversions, the other as weekly conversions per test cell⁴⁵. None of these is the platform's own documentation, and thresholds change. Check the platform's current requirements before you plan around them.

Not just the fee. Count three things.

1. Lost holdout revenue. Withholding ads from a group or region can mean sales you would otherwise have made. You pay for the answer in revenue.
2. Time. Tests need a run window and then analysis. Meanwhile your budget decision waits or goes ahead blind.
3. Team attention. Setting up regions, freezing other changes, reading the result. Small teams feel this most. One vendor guide on testing without a data team describes a manual regional holdout as the practical option⁶. That vendor sells attribution software, so read it as a view, not proof.

Set the three costs against the decision value. If the decision moves a small budget, a large test is an expensive way to be right.

You still need an answer to "what would have happened without this spend?". When a test is out, use cheaper methods. Each one has a known bias. Name it before you rely on the result.

An agency guide for small and mid-sized advertisers suggests regional pauses, blackouts and surveys⁷. Survey vendors sell "how did you hear about us" tools for exactly this job⁸. Both are interested parties. The methods are still reasonable. Just do not present them as test results.

Each method needs a baseline: the sales you would expect without the activity. Write yours down before the change. A baseline picked afterwards is a story.

Revisit the check when one of these changes:

• Volume has grown, so smaller effects become detectable.
• The budget at stake has grown, so a wrong call costs more.
• Cheaper methods keep disagreeing with each other.
• A decision is coming that a test could actually move.

That is a prompt for a conversation, not a rule. If you do test, read how to read incrementality test results before you start. The way you will read the interval should be fixed in advance.

• Which decision does this test feed, and what would we do with each result?
• Is our volume enough to see the effect size that matters?
• What does the holdout cost us in revenue, time and team attention?
• If the result is inconclusive, what then?
• What is the cheaper alternative, and what is its bias?

For ZFD: We start with what you already have. We propose a test only when its answer could change a budget and your volume can detect the effect. Not testing is also a decision. We write down what we assume instead.