Skip to content
Back to Analytics & Experimentation

Hypothesis Formation

Writing a testable hypothesis —'if we change X, then Y will increase by Z because of mechanism W' —before any code is written, so the experiment tests a prediction rather than exploring a change.

Why

An experiment without a hypothesis is exploration, not testing, and exploration experiments have no success criterion: the team ships the change if it 'looks better' (subjective, biased toward the new), reverts it if a metric moves the wrong way (which it will, by noise, 5% of the time), and learns nothing either way. A hypothesis forces the team to specify the prediction (what metric, what direction, what magnitude) and the mechanism (why the change should work), and the experiment then tests a falsifiable claim: 'if we shorten the trial from 14 to 7 days, trial-to-paid conversion will increase by 15% because users who intend to buy do so within 3 days and the longer trial dilutes intent.' If the conversion does not increase by 15%, the hypothesis is falsified, the mechanism is wrong, and the team has learned something specific (the 'dilution of intent' mechanism does not hold) rather than something vague ('shorter trials do not help'). The hypothesis also sets the sample size (powered for the predicted 15% lift, not for any detectable effect), which prevents under-powered experiments that run forever and over-powered experiments that detect noise. The most common failure is the 'let's try it' experiment —the team has a change they want to ship, they A/B test it without a hypothesis, and when the metric moves 2% they ship it, not realizing that 2% is within the noise band and the 'win' is a false positive that will revert in production.

When

Before every experiment, no exceptions. When the team wants to 'try a change' (red flag —no hypothesis). When the experiment's success criterion is unclear (red flag —no hypothesis). Before computing sample size.

How

1. State the change precisely: 'shorten trial from 14 days to 7 days' not 'make trial shorter' —the change must be specific enough that an engineer can implement it without further interpretation
2. State the primary metric and the predicted direction: 'trial-to-paid conversion will increase' not 'conversion will change' —the direction is a prediction, and an experiment that predicts 'change' is unfalsifiable (any movement confirms it)
3. State the predicted magnitude: 'increase by 15% (from 3.2% to 3.7%)' not 'increase' —the magnitude sets the sample size (powered for 15%, not for any detectable lift) and the success criterion (did it hit 15%?)
4. State the mechanism: 'because users who intend to buy do so within 3 days (data: 80% of conversions happen by day 3) and the longer trial dilutes intent by giving them 11 more days to forget/churn' —the mechanism is the 'why', and it must be grounded in data, not intuition
5. State the falsification criterion: 'if conversion does not increase by at least 10% (powered for 15%, minimum interesting effect 10%), the hypothesis is falsified and we do not ship' —this prevents the 'it moved 2%, ship it' false positive
6. State the secondary metrics and guardrails: 'secondary: revenue per trial (should not decrease); guardrail: trial cancellation rate (should not increase by more than 2%)' —these catch cases where the primary metric wins but the change hurts the business
7. State the decision: 'if the hypothesis is confirmed (conversion +15%, guardrails intact), ship to 100%; if falsified, do not ship and archive the learning' —the decision rule is set before the experiment, not after, to avoid confirmation bias
8. Write the hypothesis in a single paragraph and review it with the team before implementing: 'If we shorten the trial from 14 to 7 days, trial-to-paid conversion will increase by 15% (from 3.2% to 3.7%) because 80% of conversions happen by day 3 and the longer trial dilutes intent. If conversion does not increase by at least 10%, the hypothesis is falsified. Guardrail: trial cancellation rate must not increase by more than 2%.'
9. Register the experiment in the experiment registry with the hypothesis, the sample size, the start date, and the owner —this prevents the team from running 20 experiments and reporting only the one that 'won' (the garden-of-forking-paths problem)
10. If the team cannot write a hypothesis (step 8 fails), do not run the experiment —the change is exploration, not testing, and exploration should be shipped behind a flag and measured, not A/B tested

Example

# Hypothesis: Shorten trial from 14 to 7 days
Owner: @growth-pm  Registered: 2026-09-22  ID: EXP-247

Change: shorten trial from 14 days to 7 days (treatment group)
Control: 14-day trial (current)

Hypothesis (1 paragraph):
  If we shorten the trial from 14 to 7 days, trial-to-paid
  conversion will increase by 15% (from 3.2% to 3.7%) because
  80% of conversions happen by day 3 (data: last 90 days) and
  the longer trial dilutes intent by giving users 11 more days
  to forget or churn. If conversion does not increase by at
  least 10%, the hypothesis is falsified.

Primary metric: trial-to-paid conversion rate
  Predicted: +15% (3.2% -> 3.7%)
  Minimum interesting effect: +10% (3.2% -> 3.52%)
  Falsified if: < +10% at 95% confidence

Secondary metrics:
  - revenue per trial: should not decrease (users may buy at
    lower plan if trial is shorter)
  - time-to-first-payment: should decrease (faster decision)

Guardrails:
  - trial cancellation rate: must not increase by more than 2%
  - refund rate (first 30 days): must not increase by more than 1%
  - NPS (post-purchase): must not decrease by more than 2 points

Sample size:
  Baseline conversion: 3.2%
  MDE: 10% relative (0.32 percentage points)
  Power: 80%, alpha: 5% (two-sided)
  Required: 38,400 trials per arm (76,800 total)
  Expected duration: 18 days at 4,300 trials/day

Decision rule (set before experiment):
  - if conversion >= 3.52% AND guardrails intact: ship to 100%
  - if conversion >= 3.2% but < 3.52%: inconclusive, do not ship,
    iterate on the hypothesis (maybe 10-day trial?)
  - if conversion < 3.2%: falsified, do not ship, archive learning
  - if any guardrail breached: do not ship regardless of primary

Mechanism validation (post-experiment):
  - check if conversions shifted earlier (day 1-7 vs day 8-14)
  - if conversions just moved earlier (same total, faster), the
    mechanism is "acceleration" not "dilution reduction" and the
    hypothesis should be updated

Common Problems

  • ⚠Running 'let's try it' experiments without a hypothesis —the team A/B tests a change, the metric moves 2% (within noise), they ship it, and the 'win' reverts in production; without a predicted magnitude, any movement looks like a win and false positives ship at the 5% rate.
  • ⚠Predicting 'change' instead of a direction and magnitude —'conversion will change' is unfalsifiable (any movement confirms it), and the experiment cannot fail, which means it cannot produce a learning; the prediction must be 'increase by 15%' to be falsifiable.
  • ⚠Stating the mechanism without data —'shorter trials increase urgency' is an intuition, not a mechanism; the mechanism must be grounded in data ('80% of conversions happen by day 3'), or the hypothesis is a guess and the experiment tests a guess.
  • ⚠Setting the decision rule after the experiment —the team sees +8% (below the 10% threshold) and rationalizes '8% is still a win, let's ship it'; the decision rule must be set before the experiment to avoid confirmation bias and the garden-of-forking-paths.
  • ⚠Not registering the hypothesis before the experiment —the team runs 20 experiments, 1 'wins' at p<0.05, and they report it; without a registry, the 20 experiments are 20 independent tests and the 1 win is a false positive (the multiple-comparisons problem).

Best Practices

  • ✓Write a falsifiable hypothesis with a predicted direction, magnitude, and mechanism before any code is written —'if X, then Y increases by Z because of W' —and if the team cannot write one, the change is exploration, not testing, and should be shipped behind a flag and measured, not A/B tested.
  • ✓Ground the mechanism in data, not intuition —'80% of conversions happen by day 3' is a mechanism; 'shorter trials create urgency' is a guess, and an experiment testing a guess produces a learning only by accident.
  • ✓Set the decision rule (ship if confirmed, do not ship if falsified, guardrails must hold) before the experiment starts —this prevents confirmation bias from rationalizing a sub-threshold result into a 'win' and keeps the experiment honest.
  • ✓Register every experiment in a registry with the hypothesis, sample size, and start date before it launches —the registry prevents the multiple-comparisons problem (running 20 experiments and reporting the 1 that 'won') and makes the experiment set auditable.
  • ✓Power the experiment for the predicted magnitude (15% lift), not for any detectable effect —this prevents under-powered experiments that run forever (powered for 1%, needs 500k samples) and over-powered experiments that detect noise (powered for 0.1%, detects random fluctuation).