Skip to content
Back to Operations & SRE

Post-Incident Review

Turn every incident into systemic improvements via a blameless postmortem with tracked action items.

Why

Every incident is a free lesson in how your system fails, but only if you capture it. Without a post-incident review, the same outage recurs every quarter because the systemic cause (no deploy gate, no canary, no runbook) survives the incident — only the human who pushed the bad change gets blamed and moves on. A blameless postmortem (see blameless-postmortem) extracts the systemic cause and produces action items that fix the system so the same human mistake can't cause the same outage again. The compounding effect is dramatic: teams that do rigorous postmortems see incident counts drop 30-50% year-over-year because each fix prevents a class of future incidents, while teams that skip them see the same incidents repeat.

When

Within 5 business days of every SEV1/SEV2 resolution. Track action items to completion in sprint planning.

How

1. Schedule within 24h, hold within 5 business days
   - 60-90m; IC + responder + comms lead + a facilitator
   - While memory is fresh; pull data (alerts, deploys, chat) for the timeline

2. Build a data-driven timeline (see blameless-postmortem)
   - Minute-by-minute from logs/alerts/chat, not memory
   - "What did we know at time T?" reveals information gaps, not human errors

3. Find systemic root causes with 5 whys
   - Stop at a cause you can fix with a system change (add a test, add a gate)
   - Not a human change ("be more careful") — those don't compound

4. Write SMART action items
   - Specific, Measurable, Assigned, Realistic, Time-bound
   - "Add integration test for null-discount edge case (owner: A, due: 2w)"
   - Each prevents a class of future incidents, not just this one

5. Track to completion and measure
   - Action items are sprint work; an incident isn't closed until all items ship
   - Track "action item completion rate" and "time to close" — target >90% in 1 quarter
   - Re-incident rate: if the same root cause recurs, the action items were wrong

Example

# postmortem-2026-03-15-checkout-down.md
## Summary
Checkout 100% error for 12m; 340 orders lost (~$4.2k revenue). SEV1.

## Timeline (data-driven)
14:03  deploy v2.4.1 (canary skipped — "small change" override)
14:05  error rate 0% → 100% (null-discount bug)
14:07  alert fired → on-call acked
14:12  rollback initiated → 14:15 resolved

## Root cause (5 whys)
1. checkout broke → null-discount bug in v2.4.1
2. shipped → canary skipped
3. canary skipped → "small change" override exists in deploy tool
4. override exists → no policy preventing it for prod-critical paths
5. → fix: remove override for /checkout; require SRE approval

## Action items (SMART, tracked in sprint)
- [ ] Remove canary-skip override for prod-critical paths (owner: deploy-platform, due: 1w) [JIRA-123]
- [ ] Add integration test: null-discount edge case (owner: payments, due: 2w) [JIRA-124]
- [ ] Add deploy gate: block if canary error rate >1% (owner: deploy-platform, due: 1w) [JIRA-125]

## Metrics
- MTTD: 2m  |  MTTA: 4m  |  MTTR: 10m  |  Revenue impact: $4.2k
- Action items: 3 open, 0 done → incident NOT closed until all ship
## Blameless: engineer followed existing process. Fix the process, not the person.

Common Problems

  • ⚠Postmortem written but action items never tracked — the doc is a ritual, the systemic cause survives, and the incident recurs next quarter.
  • ⚠Action items are vague ("improve testing") with no owner or due date — they never ship, so the postmortem produced no systemic change.
  • ⚠Blameful postmortem — the engineer hides the real timeline, the systemic cause is missed, and only the human changes (temporarily).
  • ⚠No measurement of action item completion or re-incident rate — without metrics, postmortems degrade into paperwork and the improvement loop breaks.

Best Practices

  • ✓Track postmortem action items as first-class sprint work with owners and due dates; an incident is not closed until every item ships.
  • ✓Make action items SMART and systemic (add a test, add a gate) not human ("be more careful") — systemic fixes compound, human fixes don't.
  • ✓Measure action item completion rate (>90% in 1 quarter) and re-incident rate; if the same root cause recurs, the action items were wrong.
  • ✓Hold the postmortem within 5 business days blamelessly; build the timeline from data, and assume every engineer did their best with the info they had.