Alert Rules

Define alert rules that fire on user-visible symptoms, not internal noise, with clear thresholds and ownership.

Why

An alert is a contract: if it fires, a human will be woken, and that human will drop what they're doing to investigate. Every alert that fires without a user impact breaks that contract and trains engineers to ignore pages, which is how real outages get missed for 40 minutes. Good alert rules are symptom-based (the SLO is burning), not cause-based (CPU is high), because symptoms map directly to user pain and causes are often noise. The rule of thumb is: if you can't write a runbook for the alert, delete the alert — an alert without a runbook is a page that wastes 20 minutes of investigation every time.

When

After you have an SLO and a runbook. Never alert on a metric you can't tie to user impact or can't write a runbook for.

How

1. Start from the SLO, not the metric
   - "error budget burn rate > 2x over 1h" → page (symptom: users are failing)
   - NOT "CPU > 80%" (cause: might be fine, might be bad, no user signal)

2. Write the runbook before the alert
   - If you can't write "do X, then Y, then Z" for the alert, you don't understand it → don't alert
   - Runbook lives at a stable URL; the alert links to it

3. Set thresholds on burn rate, not raw values
   - Fast burn: 2x over 1h → page (catches acute outages)
   - Slow burn: 1x over 6h → ticket (catches chronic degradation without paging)
   - This gives you both fast detection and long-window coverage without dual paging

4. Tag every alert with owning team + severity
   - Route to the team that can fix it, not a generic queue
   - Severity from the impact: SEV1 (revenue down) → page; SEV3 (degraded) → ticket

5. Measure and prune alert noise monthly
   - "Alerts per on-call shift" — if >5 pages/shift, you have fatigue
   - For every alert that fired without action: delete it or convert to a ticket

Example

# Prometheus alerting rule — SLO burn rate (symptom-based)
groups:
  - name: checkout-slo
    rules:
      - alert: CheckoutSLOBurnRateFast
        expr: |
          (1 - (sum(rate(http_requests_total{service="checkout",status!~"5.."}[5m]))
               / sum(rate(http_requests_total{service="checkout"}[5m]))))
          / (1 - 0.999) > 2  # burning >2x the 99.9% SLO
        for: 2m
        labels: { severity: page, team: payments, runbook: "https://rb/checkout-slo-burn" }
        annotations:
          summary: "checkout SLO burning >2x (users failing)"
          dashboard: "https://grafana/d/checkout"

# BAD (cause-based, no user signal, will fatigue):
#   - alert: HighCPU
#     expr: cpu_usage > 0.8
# Runbook: https://rb/checkout-slo-burn
#   1. Open the dashboard link
#   2. Check error breakdown by route — which endpoint is failing?
#   3. If new deploy in last 30m → rollback (runbook: rollback-checkout)
#   4. Else → check downstream deps on the service map

Common Problems

⚠Alerting on CPU/memory >80% — high utilization is often healthy (you're getting your money's worth); alert on latency/error SLO burn instead.
⚠No runbook link in the alert payload — the on-call engineer spends 15 minutes figuring out what the alert means before they can investigate.
⚠Alerting on instantaneous values without a 'for' window — a 10-second spike pages someone at 3am for a self-resolving blip; require 'for: 2m' minimum.
⚠Routing all alerts to a generic ops queue — the team that can fix the problem never sees it, and the ops queue becomes a graveyard of ignored pages.

Best Practices

✓Alert on SLO burn rate (symptom) not resource utilization (cause) — symptoms map to user pain and rarely false-alarm.
✓Require a runbook URL in every alert rule; in code review, reject any alert rule without one — no runbook = no alert.
✓Use multi-window multi-burn-rate alerting (fast: 2x/1h pages, slow: 1x/6h tickets) to catch both acute and chronic issues without dual paging.
✓Review alert noise monthly: count pages per on-call shift, and delete or downgrade every alert that fired without user-visible action.