Statistical Significance

Definition

Statistical significance is a measure of whether an observed effect is likely to be real rather than due to random chance under a chosen false positive rate (alpha).

Example

A p-value below 0.05 suggests the result is unlikely under the null hypothesis.

How to use it

• A statistically significant result is not automatically a practically meaningful result.
• Avoid repeated peeking; it inflates false positives unless you use sequential methods.
• Pair significance with effect size and confidence intervals.
• Set alpha in advance and stick to it for the test.
• Report sample size and variance so results can be trusted and replicated.

Common mistakes

• Treating p-values as proof of causality without good test design.
• Ignoring multiple comparisons in multi-variant tests.
• Calling a result significant when the effect size is trivial.
• Changing the success metric after seeing results.

Why this matters

This term matters because it affects how you interpret performance and make budget decisions. If you use inconsistent definitions or windows, ROAS/CPA can look "better" while profit gets worse.

Practical checklist

• Write a 1-line definition for "Statistical Significance" that your team will use consistently.
• Keep the time window consistent (weekly/monthly/quarterly) when comparing trends.
• Segment results (channel/plan/cohort) before drawing big conclusions from blended averages.
• Use a calculator that references this term (e.g., A/B Test Sample Size Calculator) to sanity-check assumptions.
• Read the related guide (e.g., A/B test sample size: how to plan conversion experiments) for context and common pitfalls.

Where to use this on MetricKit