A/B test sample size: how to plan conversion experiments

A practical guide to A/B test planning: baseline CVR, MDE, alpha, power, sample size, and common pitfalls like peeking.

Updated 2026-01-28

Try it in a calculator

Example: 2.5% baseline, 0.5pp MDE, 5% alpha, 80% power

Why sample size planning matters

Without enough sample, A/B tests produce noisy results: you might ship a false win or miss a real improvement. Planning sample size sets expectations for how long a test must run.

Key inputs

Baseline CVR: your current conversion rate for the exact funnel definition.
MDE: the smallest lift worth acting on (absolute percentage points).
Alpha: tolerated false positive rate (commonly 5% for two-sided tests).
Power: probability of detecting the effect if it's real (commonly 80-90%).

Common pitfalls

Peeking and stopping early (inflates false positives).
Picking an unrealistic MDE (forces huge sample sizes).
Mixing denominators (click CVR vs session CVR) and invalidating the test.
Running tests through seasonality or major site changes without controls.

FAQ

Should I use one-sided or two-sided tests-

Two-sided is safer and more standard unless you truly would never act on a negative result. This calculator assumes a two-sided test.

What if traffic is low-

Increase MDE (test bigger changes), increase test duration, or test higher-funnel metrics first. You can also pool traffic across similar pages if the experience is consistent.

More in paid ads

CPC (Cost Per Click): definition, formula, and how to calculate

CPC explained: what cost per click means, how to calculate it, and how CPC connects to CTR, CVR, and CPA.

CPM (Cost Per 1,000 Impressions): definition, formula, and how to calculate

CPM explained: what cost per mille means, how to calculate it, and how CPM affects CPC and CPA.

CTR (Click-Through Rate): definition, formula, and how to improve

CTR explained: what click-through rate is, how to calculate it, and how to improve CTR without sacrificing conversion quality.

Activation rate: definition, formula, and how to improve activation →