Guardrail Setup

Setting guardrail metrics that automatically kill an experiment if the treatment harms the business —so a winning primary metric never comes at the cost of a degrading retention, revenue, or latency.

Why

A primary metric can win while the business loses: shortening the trial increases conversion by 15% but increases refund rate by 30% (users buy impulsively, regret, and refund), and shipping based on the primary metric alone destroys value. Guardrails are the metrics that must not degrade, and they are the safety net that prevents a 'win' from becoming a loss. The guardrails must be chosen to cover the ways the treatment can harm the business beyond the primary metric: revenue guardrails (refund rate, downgrades, churn), quality guardrails (error rate, latency, crash rate), and experience guardrails (NPS, support tickets, time-on-task). Each guardrail has a threshold (refund rate must not increase by more than 2 percentage points) and a kill switch (if breached, the experiment is stopped and the treatment is not shipped, regardless of the primary metric). The most common failure is having no guardrails —the team optimizes the primary metric, ships the winner, and discovers the refund problem two weeks later when finance complains. The second failure is having guardrails but no kill switch —the guardrail breaches, the team sees it, but they ship anyway because 'the primary metric won by 15%' and the guardrail is treated as informational rather than binding. The guardrail must be binding: if it breaches, the experiment is killed, period.

When

When defining any experiment (guardrails are mandatory). When the primary metric can be gamed at the expense of the business (conversion via shorter trial -> refund risk). Before launching any experiment to production traffic.

How

1. List the ways the treatment can harm the business beyond the primary metric: if the treatment is 'shorter trial', it can increase refunds (impulse buy regret), decrease revenue per trial (lower plan), increase support tickets (confusion), decrease NPS (rushed feeling) —each is a candidate guardrail
2. Choose 3-7 guardrails that cover revenue, quality, and experience: revenue (refund rate, churn, downgrades, revenue per user), quality (error rate, p95 latency, crash rate), experience (NPS, support tickets, time-on-task)
3. For each guardrail, set a threshold based on the pre-experiment baseline and the acceptable degradation: refund rate baseline 2.1%, acceptable increase 2 percentage points (to 4.1%) —the threshold is 4.1%, and breaching it kills the experiment
4. Make the threshold relative to the control, not absolute: 'refund rate must not exceed control by more than 2 percentage points' not 'refund rate must not exceed 4%' —the relative threshold accounts for seasonality and pre-existing trends
5. Set the kill switch: if any guardrail breaches at 95% confidence, the experiment is stopped, the treatment is not shipped, and the team is notified —the kill switch is automatic, not a human decision, to prevent 'the primary won, ship it anyway'
6. Monitor guardrails continuously during the experiment, not just at the end: check every day (or every hour for high-traffic experiments) and kill immediately if a guardrail breaches —a refund problem on day 2 should not wait until day 18
7. Choose the statistical threshold for guardrails carefully: use a lower alpha (0.01 or 0.05 one-sided) for guardrails than for the primary metric (0.05 two-sided) —guardrails are safety metrics and false negatives (missing a real degradation) are more costly than false positives (stopping a good experiment)
8. Document the guardrails in the experiment registry alongside the hypothesis: 'guardrails: refund rate (threshold: +2pp vs control), error rate (threshold: +50% vs control), NPS (threshold: -2 vs control)' —this makes the guardrails auditable and prevents the team from adding them post-hoc
9. Review guardrail breaches in the post-experiment analysis: if a guardrail breached, understand why (the mechanism that caused the harm) and update the hypothesis or the treatment before re-running —a breach is a learning, not just a kill
10. For always-on guardrails (metrics that are monitored even when no experiment is running), set the same thresholds —this catches degradations from deploys and config changes, not just experiments

Example

# Guardrails: EXP-247 (trial length 14 -> 7 days)
Primary metric: trial-to-paid conversion (+15% predicted)
Guardrails: 5 metrics, each with threshold and kill switch

Revenue guardrails:
  refund_rate_30d:
    baseline: 2.1%
    threshold: control + 2.0pp (one-sided, alpha 0.05)
    kill: if treatment > control + 2.0pp at 95% conf -> STOP
    rationale: impulse buys from shorter trial may increase regret
  revenue_per_trial:
    baseline: $7.80
    threshold: control - 5% (one-sided, alpha 0.05)
    kill: if treatment < control * 0.95 -> STOP
    rationale: users may buy lower plan if trial is shorter
  churn_30d:
    baseline: 8.2%
    threshold: control + 1.5pp (one-sided, alpha 0.05)
    kill: if treatment > control + 1.5pp -> STOP
    rationale: shorter trial may attract lower-intent users who churn

Quality guardrails:
  error_rate:
    baseline: 0.08%
    threshold: control + 50% (one-sided, alpha 0.01)
    kill: if treatment > control * 1.5 -> STOP
    rationale: trial length change should not affect error rate

Experience guardrails:
  nps_post_purchase:
    baseline: 42
    threshold: control - 2 points (one-sided, alpha 0.05)
    kill: if treatment < control - 2 -> STOP
    rationale: rushed feeling from shorter trial may hurt NPS

Monitoring:
  - checked every 6 hours during the 18-day experiment
  - if any guardrail breaches: auto-stop experiment, page @growth-pm
    and @eng-oncall, do not ship regardless of primary metric
  - guardrail results logged to experiment_guardrails table

Post-experiment guardrail results:
  refund_rate:    control 2.1%  treatment 2.3%  diff +0.2pp  OK
  rev_per_trial:  control $7.80 treatment $8.10 diff +3.8%   OK
  churn_30d:      control 8.2%  treatment 8.4%  diff +0.2pp  OK
  error_rate:     control 0.08% treatment 0.07% diff -12%    OK
  nps:            control 42    treatment 41    diff -1      OK
  -> All guardrails intact, primary metric +15.2%, SHIP

Hypothetical breach (what would have happened):
  if refund_rate had been treatment 4.5% (control 2.1%, diff +2.4pp):
  -> kill switch fires, experiment stopped on day 4
  -> treatment NOT shipped despite primary metric +18%
  -> post-mortem: impulse buy regret, iterate on 10-day trial

Common Problems

⚠No guardrails on the experiment —the team optimizes conversion, ships a 15% win, and discovers two weeks later that refund rate increased 30%; the guardrails (refund, churn, NPS) would have caught this on day 2 and killed the experiment.
⚠Guardrails without a kill switch —the guardrail breaches, the team sees it, but they ship anyway because 'the primary metric won by 15%'; the guardrail must be binding (auto-kill on breach), not informational, or it provides no protection.
⚠Absolute thresholds instead of relative —'refund rate must not exceed 4%' fires during a seasonal refund spike (holiday) even though the treatment is not causing the increase; relative thresholds (treatment vs control) account for seasonality and pre-existing trends.
⚠Checking guardrails only at the end —a refund problem on day 2 is not discovered until day 18, and 16 days of harm accrues; guardrails must be monitored continuously (every 6 hours) and kill immediately on breach.
⚠Too many guardrails with too-tight thresholds —20 guardrails at alpha 0.05 each means one will breach by chance (1 - 0.95^20 = 64% chance of at least one false breach); keep guardrails to 3-7 and use a lower alpha (0.01) for each.

Best Practices

✓Set 3-7 guardrails covering revenue (refund, churn, revenue per user), quality (error rate, latency), and experience (NPS, support tickets) for every experiment —guardrails are mandatory, not optional, because a primary metric can win while the business loses.
✓Make guardrails binding with an auto-kill switch: if any guardrail breaches at 95% confidence, the experiment is stopped and the treatment is not shipped, regardless of the primary metric —an informational guardrail that the team can override provides no protection.
✓Use relative thresholds (treatment vs control, e.g., 'refund rate must not exceed control by 2pp') not absolute —relative thresholds account for seasonality, pre-existing trends, and external shocks, and they do not fire falsely during a holiday spike.
✓Monitor guardrails continuously (every 6 hours for an 18-day experiment) and kill immediately on breach —a refund problem on day 2 should not accrue 16 days of harm before the end-of-experiment check discovers it.
✓Use a lower alpha (0.01 one-sided) for guardrails than for the primary metric (0.05 two-sided) and keep guardrails to 3-7 —guardrails are safety metrics where false negatives (missing a real degradation) are more costly than false positives, and too many guardrails at alpha 0.05 will produce false breaches by chance.