Holdout Test

Measured as

Measure Holdout Test with a fixed attribution window, conversion event, and spend basis before comparing campaigns or creative tests.

Misused when

• Peeking early and stopping on noise.
• Letting the holdout get exposed via other campaigns (invalidates the test).

Operator takeaway

• Define the holdout population and prevent spillover (contamination).
• Run long enough to cover your purchase cycle and seasonality effects.
• Use Holdout Test only inside a stable attribution rule, conversion definition, and time window so campaign comparisons stay honest.
• If performance changes, check whether the metric moved for a real business reason or because the measurement setup changed underneath you.

Next decision

• Quantify the impact with Incrementality Lift Calculator if you need to turn the definition into an operating assumption.
• Read Incrementality lift: how to compute incremental ROAS from holdouts if the decision depends on interpretation, policy, or trade-offs beyond the raw formula.