Definition
Statistical power is the probability of detecting an effect of a given size if it truly exists (1 - beta). Higher power requires larger sample sizes.
Formula
Power = 1 - beta
Example
At 80% power, you have an 80% chance to detect the target lift if it is real.
How to use it
- Typical targets are 80% or 90% power depending on decision risk.
- Higher power reduces false negatives but increases required sample size.
- Power depends on baseline rate, effect size, and variance.
- Plan power before running experiments to avoid underpowered tests.
- Use power analysis to set realistic test duration targets.
Common mistakes
- Using too low power and missing real improvements.
- Choosing an unrealistically small effect size without enough traffic.
- Stopping early before reaching the planned sample size.
- Ignoring seasonality that changes baseline conversion rates.
- Using power targets without checking data quality or bot traffic.
Measured as
Power = 1 - beta
Misused when
- Using too low power and missing real improvements.
- Choosing an unrealistically small effect size without enough traffic.
- Stopping early before reaching the planned sample size.
- Ignoring seasonality that changes baseline conversion rates.
- Using power targets without checking data quality or bot traffic.
Operator takeaway
- Typical targets are 80% or 90% power depending on decision risk.
- Higher power reduces false negatives but increases required sample size.
- Power depends on baseline rate, effect size, and variance.
- Use Statistical Power only inside a stable attribution rule, conversion definition, and time window so campaign comparisons stay honest.
- If performance changes, check whether the metric moved for a real business reason or because the measurement setup changed underneath you.
Next decision
- Quantify the impact with A/B Test Sample Size Calculator if you need to turn the definition into an operating assumption.
- Read A/B test sample size: how to plan conversion experiments if the decision depends on interpretation, policy, or trade-offs beyond the raw formula.
Where to use this on MetricKit
Calculators
- A/B Test Sample Size Calculator: Estimate sample size per variant for a conversion rate A/B test given baseline CVR, MDE, significance, and power.
Guides
- A/B test sample size: how to plan conversion experiments: A practical guide to A/B test planning: baseline CVR, MDE, alpha, power, sample size, and common pitfalls like peeking.