How to run a geo holdout test on a mid-size budget

1. Write the decision. What will you do if the channel looks highly incremental? What if it looks barely incremental? If both answers are "carry on", do not test.
2. Check the method. Is a geo test right for this question? See the next section.
3. Choose regions. Decide which are tested and which are the control, and how the control is built.
4. Size the test. Set the holdout and the run length from your own volume, not from a blog post.
5. Protect the test. Agree what is frozen in the test regions: promotions, pricing, other channel changes.
6. Run it. Do not stop early because the line looks good.
7. Read it as agreed. Use the analysis method and decision threshold you fixed in step 1.

Each step has a trap. The rest of this guide goes through them.

Use it when you cannot split people. Google's comparison of lift types says the geo-based group split is noisier and the budget requirement tends to be higher than for user-based studies¹. Meta's GeoLift documentation says the same in its own words: it recommends people-based experiments where possible, because they have higher statistical power². Geo is the method for channels and situations where regions are all you can control.

It also costs sales. Switching ads off in a region may mean orders you would have won. Price that against the decision before you start.

The same documentation calls GeoLift a research tool built by Meta's marketing science team for cases where people-based measurement is not feasible². Whichever tool you pick, someone on your side has to own the analysis.

If your question is "which method?", read geo test vs conversion lift test.

Both are ways to build the control. They are not different tests.

• Matched markets. Pair test regions with regions that look alike: similar size, similar sales pattern.
• Synthetic control. Build the control as a weighted mix of untreated regions, so it follows the test region's path before the change.

Whichever you choose, check the fit before switching anything off. The control should track the test region in the pre-test period. If it cannot, the gap after the change means little. That pre-test path is your baseline: the sales you expect without the change.

Practical filters:

• Regions with enough orders that one big customer does not decide the week.
• Regions that do not share media with the control. Think of cities that overlap on the same local audience.
• No region with a known one-off event in the test window.

Recast, a vendor that sells marketing mix modelling, ran a simulation study on synthetic data and compared GeoLift, Google's Matched Markets, CausalImpact and CausalPy³. Its point of view: the tools are not interchangeable, and they handle uncertainty differently on the same data. Synthetic data shows how methods behave. It does not rank them for your business.

Nobody can give you one number. Published advice on holdout size and duration differs between sources, and the sources are mostly vendors. So size the test from your own data.

Three things set the answer:

• Your sample size. In a geo test the unit is regions and weeks, not users.
• The minimum detectable effect (MDE): the smallest lift your design can reliably detect.
• How noisy your regional sales are around the baseline.

A vendor guide on holdout size frames the question through the effect you want to detect and the statistical power you need⁴. That is the right frame; its example figures are the vendor's, not a rule, so we do not repeat them. Another vendor, which sells attribution software, presents a manual regional holdout as the practical route for teams without a data team⁵. A practitioner's 2021 post describes simpler options: a short pause in ads, or excluding some regions⁶. All three are points of view. None of them is a threshold for your test.

Example (illustrative): Ads drive about 10% of sales in the test regions. Switching them off should cut sales there by about 10%. If weekly sales in those regions normally swing by 2% either way, a 10% drop stands out clearly. If they swing by 15% either way, the same drop is lost in the noise, and you need more regions, more weeks, or a different method. Same channel, same effect, different verdict. The noise decides.

Rule of thumb, ours: if your design cannot detect an effect big enough to change the decision, do not run it.

• Spillover. Ads aimed at the control reach people in a test region, or the other way round. Check how your channel targets locations before you choose regions.
• Seasonality. A test that spans a peak or a trough in one group of regions can look like an ad effect. The pre-test fit helps, but it cannot see next month.
• Promotions. A sale or an email send in some regions only will move sales on its own. Freeze them, or run them everywhere.
• Small markets. A handful of orders per week makes the line jump. Group small regions or leave them out.
• Other changes. New pricing, stock-outs, a new landing page. Anything that differs between groups can be mistaken for the ads.

None of these is exotic. A test that survives them is boring on purpose.

Three names come up most. They differ in method, and the choice changes the analysis, not the question you are answering.

• GeoLift. An open-source geo-experiment package in Meta's open-source repository, based on synthetic control methods⁷.
• Meridian GeoX. Google's documentation lists holdback, go-dark and heavy-up among the designs it supports⁸. Heavy-up means raising spend in some regions instead of cutting it.
• CausalImpact. A Google R package that estimates the effect of an intervention on a time series, using control series the intervention did not touch, such as non-affected markets, to predict what would have happened⁹.

For a side-by-side look, see GeoLift vs Meridian GeoX vs CausalImpact.

Read it against the control you built, and say so. State the lift and the control it is measured against.

Look at the range, not only the middle. A result can be statistically significant and still too small to matter for your budget. A result can also be a wide range that includes "no effect" and "big effect". That is "can't tell". It is a proper outcome, and a better one than a confident number from a design that could not see the effect.

Then apply the threshold you wrote in step 1. If the lower end of the range still clears it, act. If the range straddles it, decide whether more data is worth the cost or whether to use another method.

For ZFD: We want the design, the decision threshold and the analysis method signed off before the first region goes quiet. Changing them after the result is a story, not a test.