Skip to content
Back to Operations & SRE

Runbook Testing

Test runbooks in staging regularly so the first execution during a real incident isn't the first test.

Why

An untested runbook is a hypothesis — it looks right, the commands probably worked when written, but 6 months of system changes have silently broken it: the service was renamed, the DB migrated, the kubectl plugin removed. The first time you find out is during a real incident, when the on-call runs the runbook, gets an error, and now has to debug the runbook AND the incident. Testing runbooks in staging (walk the steps, or run the script with --apply) every quarter catches rot before a real incident depends on it. This is the same insight as testing code: untested code breaks in prod, untested runbooks break in incidents. The cost of a quarterly runbook test is 30 minutes; the cost of a broken runbook during a SEV1 is 15 minutes of confused debugging plus the risk of making the incident worse.

When

Quarterly for all runbooks, immediately after any system change that affects them, and before a high-stakes launch.

How

1. Schedule a quarterly runbook test day
   - Same cadence as game days; can be combined (game day tests the runbook in the scenario)
   - For each runbook: 15-30 minutes to walk through in staging

2. For doc runbooks: walk the steps manually in staging
   - Copy-paste each command; does it succeed? Does the output match the expected?
   - Click each link; does it resolve? Does the dashboard show what the runbook says?
   - Run the verification; does it actually confirm resolution?

3. For executable runbooks: run the script with --apply in staging
   - Inject the failure in staging (chaos tooling), then run the script
   - Does it detect the issue? Does it execute the fix? Does it verify resolution?
   - Log the result: pass/fail, and what broke

4. Fix or delete failing runbooks
   - A broken runbook is worse than no runbook (it misleads the on-call)
   - If the fix is small → fix in the same session; if large → ticket and owner

5. Track the pass rate as a metric
   - "Runbooks passing staging test / total" — target >90%
   - A drop in pass rate → system changes are outpacing runbook updates

Example

# Quarterly runbook test (staging)
# runbook/checkout-slo-burn.md — walk through in staging:
1. Inject the failure: deploy a broken version to staging checkout
2. Alert fires in staging (or manually trigger)
3. Open the runbook: https://rb/checkout-slo-burn
4. Step 1: kubectl rollout history → ✅ shows the broken deploy
5. Step 2: kubectl rollout undo → ✅ "deployment rolled back"
6. Verification: error rate <0.1% for 5m → ✅ confirmed
7. Result: PASS

# runbook/db-failover.md — walk through:
1. Kill the staging DB primary (ChaosMesh)
2. Step 1: promote replica → ❌ command is --promote, runbook says --take-over
3. Result: FAIL — runbook is stale
4. Action: ticket "fix db-failover.md step 1 command" (owner: @team-platform, due: 1w)

# Metric: runbooks passing / total = 18/20 = 90% ✅ (target >90%)
# The 2 failing are ticketed and will be fixed before next quarter

Common Problems

  • ⚠No staging environment that matches prod — runbook tested in staging fails in prod because the config, scale, or deps differ.
  • ⚠Test is a one-time effort — runbooks pass at write-time but rot within months as the system changes; testing must be quarterly.
  • ⚠Only testing the happy path — the runbook works when the failure is exactly as described, but real incidents have variations; test edge cases too.
  • ⚠No tracking of pass rate — without a metric, runbook rot is invisible until a real incident, which is the worst time to find it.

Best Practices

  • ✓Test every runbook in staging quarterly; walk the steps (doc) or run the script with --apply (executable) and log pass/fail.
  • ✓Keep staging close to prod (config, scale, deps) so a runbook that passes in staging passes in prod.
  • ✓Fix or delete failing runbooks in the same session; a broken runbook is worse than no runbook because it misleads the on-call.
  • ✓Track 'runbooks passing staging test / total' as a metric; target >90%, and investigate when it drops — system changes are outpacing updates.