Skip to content
Back to Operations & SRE

Runbook Versioning

Version runbooks in git alongside the code so they evolve with the system and are reviewable.

Why

A runbook in Confluence is unversioned — when someone updates it, the old version is gone, and if the update is wrong (a command typo, a deleted step), nobody can see what changed or roll back. A runbook in git is versioned: every change is a commit with an author, a message, and a diff that a reviewer can check. When the system changes, the runbook changes in the same PR, so code review catches runbook rot ('you renamed the service but didn't update the runbook'). When a runbook breaks during an incident, you can git blame to see what changed and when, and revert if needed. This is the same reason we version code: unversioned runbooks are unreviewable, unrollbackable, and silently rot — versioned runbooks are auditable, reviewable, and coupled to the system they describe.

When

When a runbook is written. Move it to git (e.g., runbooks/ in the service repo) before it goes live.

How

1. Store runbooks in git, in the service repo or a dedicated runbooks repo
   - runbooks/checkout-slo-burn.md lives in the checkout service repo
   - Or a central runbooks repo if you have many services

2. Require runbook updates in the same PR as system changes
   - "Renaming the service? Update runbooks/*.md in the same PR."
   - Code review: if a change breaks a runbook, the runbook update must be in the PR
   - This couples runbook freshness to system changes

3. Use frontmatter for metadata
   ---
   owner: @team-payments
   alert: CheckoutSLOBurnRateFast
   last_reviewed: 2026-01-15
   ---
   - Machine-readable; the audit script reads this

4. Review runbook changes in code review
   - A runbook change is a code change; reviewer checks the commands still work
   - "You changed step 2 from --take-over to --promote — did you test this in staging?"

5. Use git blame for incident forensics
   - "The runbook said --take-over but the command is --promote — when did this change?"
   - git blame runbook/db-failover.md → "Alice, 2025-12-01, 'update for new CLI version'"
   - If the change caused the incident → revert and fix

Example

# Service repo structure:
checkout-service/
  src/
  runbooks/
    checkout-slo-burn.md      # versioned with the service
    checkout-db-failover.md
  deploy/
  ...

# PR that renames the service (requires runbook update):
git diff:
  src/config.yaml:    service_name: checkout → checkout-api
  runbooks/checkout-slo-burn.md:
    - kubectl get pods -n checkout
    + kubectl get pods -n checkout-api
# Code review: "runbook updated in same PR ✅ — commands match the new name"

# Incident forensics (runbook said wrong command):
$ git blame runbooks/db-failover.md
a1b2c3d (Alice 2025-12-01)   kubectl postgres failover --take-over primary
# → Alice changed this 3 months ago; the CLI flag is now --promote
# → Revert this line, fix to --promote, add a test in the runbook audit

# Audit script reads frontmatter:
$ yq '.last_reviewed' runbooks/checkout-slo-burn.md
2026-01-15  # <90 days, fresh ✅

Common Problems

  • ⚠Runbook in Confluence — unversioned, unreviewable, and silently rots; when it breaks, you can't see what changed or roll back.
  • ⚠Runbook in a separate repo from the code — system changes and runbook updates happen in different PRs, so code review can't catch the coupling.
  • ⚠No frontmatter — the audit script can't find owner or last_reviewed, so freshness tracking is manual and doesn't happen.
  • ⚠Runbook changes not code-reviewed — a typo in a command ships silently and breaks the runbook at the worst moment.

Best Practices

  • ✓Store runbooks in git in the service repo (or a central runbooks repo) with frontmatter (owner, alert, last_reviewed).
  • ✓Require runbook updates in the same PR as system changes; code review rejects changes that break a runbook without updating it.
  • ✓Review runbook changes like code changes — a command typo is a bug; require the reviewer to confirm commands still work.
  • ✓Use git blame for incident forensics; when a runbook breaks, see what changed and when, and revert if the change caused the incident.