Skip to content
Back to Operations & SRE

Runbook Discovery

Make runbooks discoverable from the alert, the dashboard, and the service catalog — not buried in a wiki.

Why

A runbook that exists but can't be found is the same as no runbook — the on-call at 3am doesn't have time to search Confluence, guess at tags, or ask 'where do we keep runbooks?'. The runbook must be one click from the alert: the paging system surfaces the runbook_url annotation in the page, the dashboard has a 'Runbook' link in the header, and the service catalog lists runbooks per service. This three-way discoverability (alert → runbook, dashboard → runbook, catalog → runbooks) ensures the on-call finds it from wherever they are when the incident starts. Teams that wire all three report 100% of incidents start with the runbook open; teams that rely on 'search the wiki' report 30% — and the 70% that don't find it waste 10 minutes reinventing the response.

When

When a runbook is written. Wire all three discovery paths before the alert goes live.

How

1. Link the runbook from the alert (alert → runbook)
   - Alert annotation: runbook_url: "https://rb/checkout-slo-burn"
   - The paging system (PagerDuty/Opsgenie) surfaces this in the page payload
   - This is the primary path; the on-call's first action is to open the runbook

2. Link the runbook from the dashboard (dashboard → runbook)
   - Grafana dashboard header: "Runbook: checkout-slo-burn" link
   - The on-call often starts at the dashboard, not the alert; meet them there

3. List runbooks in the service catalog (catalog → runbooks)
   - Backstage/internal catalog: each service page lists its runbooks
   - This is the browse path — useful for game days, audits, and onboarding

4. Use a stable URL scheme
   - https://rb/<alert-name> or https://rb/<service>/<scenario>
   - The URL should be guessable; the on-call can construct it if the link is missing

5. Make runbooks searchable by symptom, not just alert name
   - "checkout error rate high" should find the SLO burn runbook
   - Tag runbooks with user-visible symptoms, not just internal alert names

Example

# Alert rule (Prometheus) — runbook_url annotation
- alert: CheckoutSLOBurnRateFast
  expr: ...
  annotations:
    runbook_url: "https://rb/checkout-slo-burn"
    dashboard: "https://grafana/d/checkout"

# Grafana dashboard — runbook link in the header
# (dashboard JSON):
"links": [
  { "title": "Runbook: SLO burn", "url": "https://rb/checkout-slo-burn", "type": "link" },
  { "title": "Alert: CheckoutSLOBurnRateFast", "url": "...", "type": "link" }
]

# Service catalog (Backstage) — runbooks per service
# catalog-info.yaml:
apiVersion: backstage.io/v1alpha1
kind: Component
metadata:
  name: checkout
  links:
    - url: https://rb/checkout-slo-burn
      title: "Runbook: SLO burn"
    - url: https://rb/checkout-db-failover
      title: "Runbook: DB failover"

# On-call's experience: alert fires → page includes runbook_url → one click → runbook open.
# If they start at the dashboard → "Runbook" link in header → same URL.
# If they're browsing the catalog → checkout page → runbooks listed.

Common Problems

  • ⚠Runbook only linked from the alert — if the on-call starts at the dashboard (common), they don't find it and reinvent the response.
  • ⚠Runbook URL is a Confluence deep link that changes when the page is moved — the alert's link 404s silently and nobody notices.
  • ⚠Runbooks tagged by alert name only — searching 'checkout error rate high' finds nothing because the runbook is tagged 'CheckoutSLOBurnRateFast'.
  • ⚠No service catalog — runbooks are scattered across repos and wikis, so the browse path (useful for game days and audits) doesn't exist.

Best Practices

  • ✓Wire all three discovery paths: alert annotation (runbook_url), dashboard header link, and service catalog listing — meet the on-call where they are.
  • ✓Use a stable, guessable URL scheme (https://rb/<alert-name>) hosted in git, not a Confluence link that rots when the page is moved.
  • ✓Tag runbooks with user-visible symptoms ('checkout error rate high'), not just alert names, so search works from the on-call's mental model.
  • ✓List runbooks per service in the service catalog (Backstage) so the browse path exists for game days, audits, and onboarding.