Definition
A holdout test withholds ads from a control group and compares outcomes to measure incremental lift.
How to use it
- Define the holdout population and prevent spillover (contamination).
- Run long enough to cover your purchase cycle and seasonality effects.
Common mistakes
- Peeking early and stopping on noise.
- Letting the holdout get exposed via other campaigns (invalidates the test).
Why this matters
This term matters because it affects how you interpret performance and make budget decisions. If you use inconsistent definitions or windows, ROAS/CPA can look "better" while profit gets worse.
Practical checklist
- Write a 1-line definition for "Holdout Test" that your team will use consistently.
- Keep the time window consistent (weekly/monthly/quarterly) when comparing trends.
- Segment results (channel/plan/cohort) before drawing big conclusions from blended averages.
- Use a calculator that references this term (e.g., Incrementality Lift Calculator) to sanity-check assumptions.
- Read the related guide (e.g., Incrementality lift: how to compute incremental ROAS from holdouts) for context and common pitfalls.
Where to use this on MetricKit
Calculators
- Incrementality Lift Calculator: Estimate incremental conversions, incremental ROAS, and incremental profit from a holdout test.
Guides
- Incrementality lift: how to compute incremental ROAS from holdouts: Turn an exposed vs holdout test into incremental conversions, incremental ROAS, and incremental profit for decision-making.
- Incrementality: how to tell if ads are actually driving growth: Platform-reported ROAS can overstate impact. Learn what incrementality means, when it matters, and practical ways to test it.
- Paid ads measurement hub: ROAS, MER, marginal ROAS, and incrementality: A practical hub for paid ads measurement: connect ROAS to profit, use MER for top-down truth, watch marginal ROAS for scale, and validate incrementality with holdouts.
- Attribution vs incrementality: what to trust, when, and how to test: A practical guide to attribution vs incrementality: common attribution models, window pitfalls, how MER/marginal ROAS fit in, and how to run holdout/geo tests.