Geo Holdout

Definition

A geo holdout is an incrementality test that turns ads off in selected geographies and compares outcomes versus similar geographies where ads remain on.

Example

You pause ads in two matched regions and compare sales to two regions where ads continue.

How to use it

• Match geos by baseline demand and seasonality; otherwise results are noisy.
• Use longer windows for longer sales cycles or higher conversion lag.
• Control for distribution changes, pricing changes, or competitor shocks.
• Use multiple matched pairs to reduce volatility from local events.
• Confirm media weight is large enough to create a measurable delta.
• Run a pre-period to confirm the geos track closely before the test.

Common mistakes

• Using too few regions and over-interpreting noise.
• Choosing geos with different baseline demand patterns.
• Letting organic or PR spikes in a single region distort outcomes.
• Changing budgets outside the test and masking the true lift.

Why this matters

This term matters because it affects how you interpret performance and make budget decisions. If you use inconsistent definitions or windows, ROAS/CPA can look "better" while profit gets worse.

Practical checklist

• Write a 1-line definition for "Geo Holdout" that your team will use consistently.
• Keep the time window consistent (weekly/monthly/quarterly) when comparing trends.
• Segment results (channel/plan/cohort) before drawing big conclusions from blended averages.
• Use a calculator that references this term (e.g., Incrementality Lift Calculator) to sanity-check assumptions.
• Read the related guide (e.g., Attribution vs incrementality: what to trust, when, and how to test) for context and common pitfalls.

Where to use this on MetricKit