Stop Condition

Pre-defined rules for when to stop a test —sample size reached, guardrail breached, or max duration —so stopping is not a judgment call made mid-test.

Why

The decision to stop a test is the decision to ship or kill, and if it is made by looking at the result and deciding 'this looks good enough', it is p-hacking. Every day you peek at the p-value and consider stopping, you inflate the false-positive rate, because you are running multiple implicit tests (one per peek) and counting the one that crossed the threshold. The stop condition removes this by pre-defining the stopping rule: the test stops when the pre-committed sample size is reached, OR when a guardrail breaches its stop threshold, OR when the max duration is hit. None of these depend on the primary metric's p-value, so none of them introduce optional stopping bias. This is what makes the test's result trustworthy: the stopping rule was fixed before the result existed, so the result cannot have influenced the stopping.

When

Defined during test design, before traffic. Evaluated continuously (guardrails) and at sample-size milestone (primary metric).

How

1. Pre-commit three stop conditions in the experiment registry before exposure:
   a. Sample-size stop: stop when both arms reach the pre-computed n_per_arm
   b. Guardrail stop: stop immediately if any guardrail breaches 2x its threshold (checked hourly)
   c. Max-duration stop: stop at the pre-committed max duration (e.g., 14 days) regardless of sample size, to prevent indefinite tests
2. Do NOT add a "significance stop" —stopping when p<0.05 is optional stopping and inflates false positives from 5% to 30%+
3. Implement the stop conditions in the experiment platform, not in a human checklist —the platform stops the test automatically when a condition is met, removing the temptation to override
4. For guardrail stops: route an alert to the on-call and the experiment owner, freeze the variant (stop exposing new users), and keep existing users in their bucket for analysis
5. For sample-size stop: this is the planned end. Compute the final result, apply the decision rule, and ship/kill per the pre-committed rule
6. For max-duration stop: the test did not reach significance. Conclude "inconclusive at this sample size" (not "no effect"), and decide whether to re-run with more traffic or kill
7. Document which stop condition fired and why —this is the audit trail that shows the test was not stopped by peeking
8. Never override a stop condition without a documented reason and sign-off —overriding "the guardrail stop because we think it's noise" is how harm ships

Example

# Experiment #447 stop conditions (pre-committed)
Registered: 2026-09-20  Owner: @pm-checkout

1. Sample-size stop: 45,289 per arm (computed for 5% MDE, 80% power)
   -> expected at day 4 (12000 impressions/day)
2. Guardrail stop (checked hourly):
   - page_load_p95 > 1.4s (baseline 1.2s + 200ms stop threshold)
   - error_rate > 1.8% (baseline 0.8% + 1.0pp stop threshold)
   - checkout_conversion < 62.7% (baseline 64% - 2% rel stop)
   -> any breach: freeze variant, alert on-call + owner
3. Max-duration stop: day 7 (sample-size + 3 days slack)
   -> if reached without sample-size, conclude inconclusive

NOT a stop condition:
   - p < 0.05 on primary metric (this is optional stopping, forbidden)

Execution log:
  Day 1: n=24k/arm, guardrails OK
  Day 2: n=24k/arm, guardrails OK
  Day 3: n=36k/arm, page_load_p95=1.45s -> GUARDRAIL STOP
    -> variant frozen, alert sent, test stopped at 14:12
    -> root cause: BlueCta CSS regression, fixed, re-run as #448
  Re-run #448:
  Day 4: n=45.3k/arm -> SAMPLE-SIZE STOP (planned end)
    -> CTR +6.2% (p=0.003), all guardrails OK -> SHIP

Audit: both stops were pre-committed conditions, not peek-based.
No optional stopping bias; result is trustworthy.

Common Problems

⚠Stopping when p<0.05 —this is optional stopping and it is the most common stop-condition error; it inflates false positives from 5% to 30%+, meaning a third of shipped 'wins' are noise.
⚠No max-duration stop —the test runs for 6 weeks because the sample size is never reached (traffic was over-estimated), the team forgets about it, and it is left running with stale code that blocks other experiments.
⚠Guardrail stop without freezing the variant —the test 'stops' but new users keep getting exposed to the treatment, harm continues, and the 'stop' was just a label; the variant must be frozen, not just the monitoring.
⚠Overriding the stop condition because 'the result looks almost significant' —the team extends the test by 3 days to 'get significance', which is optional stopping by another name, and the resulting significance is meaningless.

Best Practices

✓Pre-commit all three stop conditions (sample-size, guardrail, max-duration) in the registry before exposure —the pre-commitment is what makes the result valid; without it, every stop is a peek.
✓Implement stop conditions in the platform, not in a human checklist —the platform stops automatically, removing the temptation to override; a human who can override is a human who will override under deadline pressure.
✓Always include a max-duration stop (e.g., 2x the expected duration) —this prevents indefinite tests that block traffic, and it forces a decision on underpowered tests rather than letting them drift.
✓When a guardrail stop fires, freeze the variant immediately (stop new exposures) but keep existing users in their bucket for analysis —this limits harm to new users while preserving the data needed to diagnose the breach.