Definition
Statistical significance is a measure of whether an observed effect is likely to be real rather than due to random chance under a chosen false positive rate (alpha).
Example
A p-value below 0.05 suggests the result is unlikely under the null hypothesis.
How to use it
- A statistically significant result is not automatically a practically meaningful result.
- Avoid repeated peeking; it inflates false positives unless you use sequential methods.
- Pair significance with effect size and confidence intervals.
- Set alpha in advance and stick to it for the test.
- Report sample size and variance so results can be trusted and replicated.
Common mistakes
- Treating p-values as proof of causality without good test design.
- Ignoring multiple comparisons in multi-variant tests.
- Calling a result significant when the effect size is trivial.
- Changing the success metric after seeing results.
Measured as
Measure Statistical Significance with a fixed attribution window, conversion event, and spend basis before comparing campaigns or creative tests.
Misused when
- Treating p-values as proof of causality without good test design.
- Ignoring multiple comparisons in multi-variant tests.
- Calling a result significant when the effect size is trivial.
- Changing the success metric after seeing results.
Operator takeaway
- A statistically significant result is not automatically a practically meaningful result.
- Avoid repeated peeking; it inflates false positives unless you use sequential methods.
- Pair significance with effect size and confidence intervals.
- Use Statistical Significance only inside a stable attribution rule, conversion definition, and time window so campaign comparisons stay honest.
- If performance changes, check whether the metric moved for a real business reason or because the measurement setup changed underneath you.
Next decision
- Quantify the impact with A/B Test Sample Size Calculator if you need to turn the definition into an operating assumption.
- Read A/B test sample size: how to plan conversion experiments if the decision depends on interpretation, policy, or trade-offs beyond the raw formula.
Where to use this on MetricKit
Calculators
- A/B Test Sample Size Calculator: Estimate sample size per variant for a conversion rate A/B test given baseline CVR, MDE, significance, and power.
Guides
- A/B test sample size: how to plan conversion experiments: A practical guide to A/B test planning: baseline CVR, MDE, alpha, power, sample size, and common pitfalls like peeking.