Ship Decision

Why

When

How

1. Wait for the experiment to reach its pre-registered sample size —do not peek at the result early and stop the experiment when it 'looks significant', because peeking inflates the false-positive rate (each peek is a independent test, and 10 peeks at alpha 0.05 produce a 40% false-positive rate)
2. Check all guardrails first: if any guardrail breached, the experiment is killed, the treatment is not shipped, and the team investigates why the guardrail breached (the mechanism that caused the harm) —guardrails are binding, the primary metric does not override them
3. Apply the pre-registered decision rule: if conversion >= 3.52% (the minimum interesting effect), ship; if 3.2% <= conversion < 3.52%, inconclusive, iterate; if conversion < 3.2%, falsified, abandon —the rule was set before the experiment to avoid confirmation bias
4. Examine segment-level effects, not just the average: the +8% average may be +20% for new users and -4% for existing users —if so, ship to the new-user segment (segment-specific ship) rather than shipping or abandoning based on the average
5. Check the effect size against the winner's curse: the experimental estimate is biased upward (you are shipping because it won, and the winner is an overestimate); expect the production effect to be 60-80% of the experimental estimate, and verify this in the first 2 weeks of production
6. If the decision is 'ship to 100%': roll out gradually (10% -> 50% -> 100% over a week), monitor guardrails during the rollout, and auto-rollback if any guardrail breaches in production
7. If the decision is 'iterate': the result was inconclusive (not a win, not a falsification), and the team should try a variant (10-day trial instead of 7-day) with a new hypothesis —the original experiment produced a learning (the 7-day trial did not produce a 15% lift) that informs the next experiment
8. If the decision is 'abandon': the hypothesis was falsified, the mechanism does not hold, and the team documents the learning ('shorter trials do not increase conversion via the dilution-of-intent mechanism; 80% of conversions already happen by day 3, so the trial length is not the binding constraint') —this prevents the team from re-trying the same hypothesis in 6 months
9. Document the experiment in the archive: hypothesis, result, guardrail results, segment effects, decision, and learning —the archive is searchable, so future experiments can reference it and the team does not re-run falsified hypotheses
10. Verify the production effect 2 weeks after shipping: compare the production metric to the experimental estimate; if the production effect is < 50% of the estimate, the winner's curse was severe and the experiment's value is less than predicted —this calibrates future experimental estimates

Example

# Ship decision: EXP-247 (trial length 14 -> 7 days)
Sample size reached: day 18, 38,400 per arm (76,800 total)
Pre-registered decision rule:
  conversion >= 3.52% AND guardrails intact -> ship to 100%
  3.20% <= conversion < 3.52% -> inconclusive, iterate
  conversion < 3.20% -> falsified, abandon

Step 1: Guardrail check (all must pass)
  refund_rate:    control 2.1%  treatment 2.3%  +0.2pp  OK (< +2pp)
  rev_per_trial:  control $7.80 treatment $8.10 +3.8%   OK (> -5%)
  churn_30d:      control 8.2%  treatment 8.4%  +0.2pp  OK (< +1.5pp)
  error_rate:     control 0.08% treatment 0.07% -12%    OK (< +50%)
  nps:            control 42    treatment 41    -1      OK (> -2)
  -> All guardrails intact, proceed to primary metric

Step 2: Primary metric
  control conversion:    3.18% (1,221 / 38,400)
  treatment conversion:  3.66% (1,405 / 38,400)
  relative lift:         +15.1% (95% CI: +8.2% to +22.0%)
  p-value:               0.002 (significant at alpha 0.05)
  -> 3.66% >= 3.52% threshold, SHIP to 100%

Step 3: Segment analysis (before shipping)
  Segment          | Control | Treatment | Lift   | Decision
  -----------------|---------|-----------|--------|----------
  new users        | 2.8%    | 3.5%      | +25%   | ship
  returning users  | 4.2%    | 4.3%      | +2%    | neutral
  mobile           | 2.1%    | 2.9%      | +38%   | ship
  desktop          | 3.8%    | 4.1%      | +8%    | ship
  -> No segment is negative, ship to 100% (not segment-specific)

Step 4: Gradual rollout
  Day 1:  10% of traffic (monitor guardrails)
  Day 3:  50% of traffic (monitor guardrails)
  Day 7: 100% of traffic (monitor guardrails)
  Auto-rollback if any guardrail breaches in production

Step 5: Production verification (2 weeks post-ship)
  Experimental estimate: +15.1%
  Production effect (day 14): +11.2%
  Winner's curse ratio: 11.2 / 15.1 = 74% (expected 60-80%)
  -> Production effect confirmed, experiment was not overestimated

Step 6: Archive
  EXP-247: hypothesis confirmed, shipped to 100% on Oct 10.
  Learning: shorter trial increases conversion via reduced
  dilution of intent; effect is larger for new and mobile users.
  Production effect 74% of experimental estimate (calibration
  for future experiments: expect ~75% of experimental lift).

Common Problems

• ⚠Peeking at the result early and stopping when it 'looks significant' —each peek is an independent test, and 10 peeks at alpha 0.05 produce a 40% false-positive rate; the experiment must run to its pre-registered sample size, and peeking invalidates the p-value.
• ⚠Rationalizing a sub-threshold result —the team sees +8% (below the 10% threshold) and ships it because '8% is still a win'; the pre-registered decision rule exists to prevent this, and overriding it post-hoc is confirmation bias.
• ⚠Shipping on the average without checking segments —the +8% average is +20% for new users and -4% for existing; shipping to 100% hurts existing users, and the right decision is to ship to the new-user segment only.
• ⚠Not verifying the production effect —the experimental estimate is +15% but the production effect is +5% (winner's curse); without a 2-week production check, the team believes the experiment delivered 15% and over-invests in similar experiments.
• ⚠Not documenting the learning when abandoning —the team falsifies a hypothesis, abandons the change, and does not write it down; 6 months later, a new team member re-runs the same experiment and wastes a quarter re-learning what was already known.

Best Practices

• ✓Wait for the pre-registered sample size before checking the result —peeking early and stopping when 'significant' inflates the false-positive rate to 40% (10 peeks at alpha 0.05), and the experiment must run to completion to produce a valid p-value.
• ✓Apply the pre-registered decision rule (ship if >= threshold, iterate if inconclusive, abandon if falsified) without post-hoc override —the rule was set before the experiment to prevent confirmation bias, and rationalizing a sub-threshold result into a 'win' is the bias the rule exists to prevent.
• ✓Examine segment-level effects before shipping to 100% —the average may hide a +20% segment and a -4% segment, and the right decision may be to ship to the positive segment only rather than shipping or abandoning based on the average.
• ✓Verify the production effect 2 weeks after shipping and compare to the experimental estimate —the winner's curse means the production effect is typically 60-80% of the estimate, and if it is < 50%, the experiment was overestimated and future estimates should be calibrated down.
• ✓Document every experiment (including abandoned ones) in a searchable archive with the hypothesis, result, and learning —the archive prevents the team from re-running falsified hypotheses and makes the accumulated learning available to future team members.