Skip to content
Back to Operations & SRE

Blameless Postmortem

Run blameless postmortems that find systemic causes and produce action items, not scapegoats.

Why

A blameful postmortem produces lies and fear: the engineer who pushed the bad deploy hides it, the next incident goes under-reported, and the systemic cause (no deploy gate, no canary) survives to cause another outage. A blameless postmortem assumes every engineer was doing their best with the information they had, and the failure is in the system — the tooling that allowed a bad deploy to reach production, the alert that didn't fire, the runbook that didn't exist. This produces honest timelines, real root causes, and action items that fix the system so the same human mistake can't cause the same outage again. Google's data is clear: blameless cultures have fewer repeat incidents because the systemic fixes compound, while blameful cultures have the same incident every quarter because only the human changed.

When

After every SEV1 and SEV2 incident, within 5 business days. SEV3 optional; SEV4 no postmortem.

How

1. Schedule within 24h of resolution, hold within 5 business days
   - While memory is fresh; include the on-call, the author of the change, and a facilitator
   - 60-90 minutes; invite only those with direct knowledge (not the whole org)

2. Build the timeline from data, not memory
   - Pull alerts, deploys, chat logs, metric screenshots; assemble a minute-by-minute timeline
   - "What did we know at time T?" — the timeline reveals information gaps, not human errors

3. Find systemic causes with "5 whys"
   - "The deploy broke checkout" → "why did it pass CI?" → "no integration test for X" → "why no test?" → ...
   - Stop at a cause you can fix with a system change (add a test, add a gate), not a human change ("be more careful")

4. Write action items that fix the system
   - Each action item: owner, due date, and the systemic fix (not "be more careful")
   - Examples: "add integration test for discount edge case (owner: A, due: 2w)", "add deploy gate blocking >50% error rate (owner: B, due: 1w)"

5. Track action items to completion
   - Postmortem is not done when the doc is written; it's done when all action items ship
   - Review open action items in sprint planning; an incident with open items is not closed

Example

# postmortem-2026-03-15-checkout-down.md
## Summary
Checkout error rate hit 100% for 12 minutes; 340 orders lost (~$4.2k revenue).

## Timeline (from data)
- 14:03  Deploy v2.4.1 shipped (canary skipped — "small change")
- 14:05  Error rate 0% → 100% (new code threw on null discount)
- 14:07  Alert CheckoutSLOBurnRateFast fired
- 14:09  On-call acked, opened runbook
- 14:12  Rollback initiated
- 14:15  Error rate 0%, incident resolved

## Root cause (5 whys)
1. Why did checkout break? → new code threw on null discount
2. Why did it ship? → canary was skipped
3. Why was canary skipped? → "small change" override exists in the deploy tool
4. Why does the override exist? → no policy preventing it for prod-critical paths
5. → Systemic fix: remove the canary-skip override for /checkout; require explicit SRE approval

## Action items (systemic, owned, dated)
- [ ] Remove canary-skip override for prod-critical paths (owner: deploy-platform, due: 1w)
- [ ] Add integration test: null discount edge case (owner: payments, due: 2w)
- [ ] Add deploy gate: block if error rate >1% in canary (owner: deploy-platform, due: 1w)
## Blameless note: the engineer who skipped canary followed an existing, documented process. The fix is the process, not the person.

Common Problems

  • ⚠Postmortem blames a person ("engineer X pushed a bad deploy") instead of the system that allowed it — the same mistake recurs because only the human changed.
  • ⚠Action items are vague ("be more careful", "improve testing") with no owner or due date — they never ship, so the systemic cause survives.
  • ⚠Postmortem written but action items never tracked — the doc is a ritual, not a fix; incidents repeat because the system was never changed.
  • ⚠Skipping postmortem for "small" incidents — small incidents are early warnings of systemic issues; skipping them guarantees a big one later.

Best Practices

  • ✓Assume every engineer did their best with the information they had; the failure is in the system — this produces honest timelines and systemic fixes.
  • ✓Require every action item to have an owner, a due date, and a system change (not "be more careful"); track items to completion in sprint planning.
  • ✓Hold the postmortem within 5 business days while memory is fresh, and build the timeline from data (logs, alerts, deploys), not recollection.
  • ✓Track postmortem action items as first-class sprint work; an incident is not closed until every systemic fix ships, not when the doc is written.