Definition
An A/B test compares two variants (A and B) to measure whether a change improves an outcome (e.g., conversion rate).
Example
Variant A converts at 2.0% and Variant B at 2.3%; the test checks if the lift is real.
How to use it
- Define a primary metric and a fixed test duration/sample size before starting.
- Avoid peeking and stopping early based on noisy intermediate results.
- Randomize exposure so groups are comparable.
- Run the test long enough to cover conversion lag.
- Use the same traffic allocation rules across variants to avoid bias.
- Keep landing pages and offers consistent so only one variable changes.
Common mistakes
- Changing multiple variables at once and losing causal clarity.
- Declaring a winner without enough sample size.
- Letting traffic allocation drift mid-test.
- Running overlapping tests that contaminate the same audience.
- Switching metrics after seeing early results.
Measured as
Measure A/B Test with a fixed attribution window, conversion event, and spend basis before comparing campaigns or creative tests.
Misused when
- Changing multiple variables at once and losing causal clarity.
- Declaring a winner without enough sample size.
- Letting traffic allocation drift mid-test.
- Running overlapping tests that contaminate the same audience.
- Switching metrics after seeing early results.
Operator takeaway
- Define a primary metric and a fixed test duration/sample size before starting.
- Avoid peeking and stopping early based on noisy intermediate results.
- Randomize exposure so groups are comparable.
- Use A/B Test only inside a stable attribution rule, conversion definition, and time window so campaign comparisons stay honest.
- If performance changes, check whether the metric moved for a real business reason or because the measurement setup changed underneath you.
Next decision
- Quantify the impact with A/B Test Sample Size Calculator if you need to turn the definition into an operating assumption.
- Read A/B test sample size: how to plan conversion experiments if the decision depends on interpretation, policy, or trade-offs beyond the raw formula.
Where to use this on MetricKit
Calculators
- A/B Test Sample Size Calculator: Estimate sample size per variant for a conversion rate A/B test given baseline CVR, MDE, significance, and power.
Guides
- A/B test sample size: how to plan conversion experiments: A practical guide to A/B test planning: baseline CVR, MDE, alpha, power, sample size, and common pitfalls like peeking.