Skip to content
Back to Analytics & Experimentation

Statistical Significance

Computing and interpreting p-values and confidence intervals so you know whether a 3% lift is real or noise —the gate between 'run another experiment' and 'ship it'.

Why

Every metric fluctuates. When you run an A/B test and see a 3% lift, the single most important question is: is this 3% a signal that the feature works, or is it noise that would have happened anyway? Statistical significance is the formal answer to that question, and getting it wrong is expensive in both directions. If you call noise significant (false positive), you ship a feature that does nothing and clutter the product. If you call a real effect non-significant (false negative), you kill a feature that would have worked and your competitor ships it. The 0.05 threshold is a convention, not a law —it means a 1-in-20 chance of a false positive, which is acceptable when you run 20 experiments a year but unacceptable when you run 200. Understanding the actual false-positive rate of your testing program (which rises with the number of tests) is what separates a mature analytics practice from one that ships 30% illusions.

When

At the end of every A/B test, before the ship/kill decision. Never mid-test (peeking inflates false positives).

How

1. Before the test, compute the minimum detectable effect (MDE) and required sample size —if you cannot reach the sample size, do not run the test (you will get a non-significant result regardless and waste the traffic)
2. Run the test to the pre-committed sample size, no early stopping —peeking at p-values mid-test and stopping when p<0.05 inflates the false-positive rate from 5% to 30%+
3. Compute the effect size (lift) and its 95% confidence interval —the interval tells you both "is it significant" (does it exclude 0) and "how big is it" (the range of plausible values)
4. Report the p-value, the confidence interval, and the effect size together —a p=0.04 with a CI of [0.1%, 5.9%] is a significant result where the effect might be trivially small, and the CI reveals this
5. Apply a multiple-comparison correction if you tested more than one metric or variant —Bonferroni (divide alpha by number of tests) is conservative, Benjamini-Hochberg (FDR) is less so; both beat the uncorrected approach
6. Distinguish statistical significance (p<0.05) from practical significance (lift > business-relevant threshold) —a 0.1% lift can be p<0.001 with enough traffic, and it is still not worth shipping if the business threshold is 1%
7. Document the result in the experiment registry: hypothesis, sample size, actual lift, CI, p-value, decision —this is the audit trail that prevents re-running the same test and counting the significant one
8. If the result is non-significant, do not conclude "no effect" —conclude "we cannot detect an effect at this sample size"; the effect may be real but smaller than your MDE

Example

# Experiment: new onboarding flow vs control
Hypothesis: new flow increases 7-day activation by >= 2pp
MDE: 2pp  Alpha: 0.05  Power: 0.80
Required sample: 12,400 per arm (computed via two-proportion z-test)
Actual sample: 12,450 treatment, 12,380 control (reached)

Results:
  Control activation:     21.4%  (2650/12380)
  Treatment activation:   23.1%  (2876/12450)
  Lift: +1.7pp  (+7.9% relative)

  95% CI on lift: [0.3pp, 3.1pp]  (excludes 0 -> significant)
  p-value: 0.018  (< 0.05)

Multiple comparisons: 1 primary metric, 2 guardrails checked
  Bonferroni alpha: 0.05/3 = 0.0167
  p=0.018 > 0.0167 -> NOT significant after correction

Decision: DO NOT SHIP
  The result is nominally significant (p=0.018) but fails the
  Bonferroni-corrected threshold (0.0167) because we checked 3 metrics.
  Even without correction, the lower CI bound (0.3pp) is below the
  pre-committed MDE (2pp), so the effect may be smaller than we
  designed the test to detect. Re-run with 4x sample to resolve.

Common Problems

  • ⚠Peeking at the p-value mid-test and stopping when it crosses 0.05 —this is the single most common error and it inflates false positives from 5% to 30%+, meaning a third of your 'wins' are noise that you shipped.
  • ⚠Testing 20 metrics, finding one with p<0.05, and reporting it as a win —with 20 tests at alpha=0.05, the expected number of false positives is 1, so the one 'win' is exactly what you would expect by chance, and it is almost certainly noise.
  • ⚠Conflating 'not significant' with 'no effect' —a non-significant result means 'we cannot detect an effect at this sample size', not 'the effect is zero'; underpowered tests kill real features by reporting non-significance.
  • ⚠Using 0.05 as a universal threshold regardless of test volume —at 200 tests/year, 0.05 gives 10 expected false positives, which is a feature-a-quarter of pure illusion; the threshold must scale with the testing program.

Best Practices

  • ✓Pre-commit the sample size and analysis plan before the test starts, and write it down —this is what makes peeking impossible, because the decision rule is fixed before there is a result to peek at.
  • ✓Apply multiple-comparison correction (Benjamini-Hochberg FDR for exploration, Bonferroni for confirmatory) whenever more than one metric or variant is tested —the correction is the difference between a 5% false-positive rate and a 40% one.
  • ✓Report the confidence interval, not just the p-value —the CI shows the range of plausible effects, which is what the business needs to decide 'is this worth shipping', whereas p<0.05 only says 'not zero'.
  • ✓Calibrate the false-positive rate of your testing program by periodically running A/A tests —if 5% of A/A tests show p<0.05, your pipeline is healthy; if 15% do, you have a bug in randomization, bot contamination, or peeking.