Skip to content
Back to Operations & SRE

Game Day

Run game days that inject failures into staging to practice incident response before a real outage.

Why

The first time someone is Incident Commander should not be during a real SEV1 — they'll be learning the role while revenue is down, which is the worst possible time to learn. A game day injects a planned failure into staging (kill a DB, saturate a cache, deploy a broken version) and runs the full incident response: IC designation, war room, comms, rollback, postmortem. This trains the IC muscle, tests the runbook (does the rollback actually work?), and surfaces gaps in monitoring or tooling in a safe environment. Teams that run quarterly game days report 50% faster MTTR on real incidents because the response is rehearsed, and they catch runbook rot (the rollback script that broke 3 months ago) before a real outage depends on it.

When

Quarterly for Tier 1 services, before major launches, and whenever a new on-call engineer joins the rotation.

How

1. Pick a failure scenario relevant to your top risks
   - "DB primary fails" if you're single-DB; "region down" if multi-region; "bad deploy" if you ship often
   - The scenario should be plausible — you're practicing for a real outage, not a sci-fi edge case

2. Inject the failure in staging at a scheduled time
   - Use chaos tooling (ChaosMesh, Gremlin) or manual: kill the pod, saturate the cache, deploy the broken branch
   - Notify the team: "game day starting, staging checkout will be down for ~30m"

3. Run the full incident response, for real
   - Designate IC, open war room, run comms, follow the runbook, attempt rollback
   - The only difference from a real incident is that it's staging and scheduled — everything else is real

4. Debrief immediately after
   - Did the alert fire? Did the runbook work? Did the rollback succeed? Where did the response stall?
   - Capture action items: "runbook step 3 is wrong", "alert didn't fire for 8m", "IC didn't know who the SME was"

5. Track game-day action items like postmortem items
   - Fix the runbook, fix the alert, train the IC — these are real reliability wins, not practice
   - The game day that finds 3 broken runbooks is a huge success; the one that finds nothing is a worry

Example

# Game day: "DB primary fails" for checkout service (staging)
13:55  Facilitator: "Game day starting in 5m. Scenario: DB primary failure.
       IC will be @new-oncall-engineer (first time as IC). Staging only."
14:00  Facilitator kills the DB primary in staging (ChaosMesh pod-kill)
14:02  Alert fires (DbPrimaryDown) → on-call acks, opens #gameday
14:03  New IC: "I'm IC. @debugger, check the runbook for DB failover.
       @comms, draft a status page update (staging)."
14:08  Debugger: "Runbook step 3 says 'promote replica' but the command is wrong —
       the flag is --promote not --take-over. Runbook is stale."
14:10  IC: "Decision: fix the runbook command inline, proceed with failover."
14:15  Failover complete, checkout recovered in staging.
14:20  Debrief:
       - Action 1: fix runbook step 3 command (owner: @debugger, due: 1w)
       - Action 2: alert fired in 2m — good, no action
       - Action 3: IC did well for first time; add "who is SME" to IC checklist
# Found a stale runbook in 20 minutes, in staging, with zero user impact.

Common Problems

  • ⚠Game day scenario is too tame — 'restart a pod' tests nothing; pick a failure that would actually be a SEV1 in prod.
  • ⚠Team treats it as a drill, not a real incident — no IC, no war room, no comms; then the real incident reveals the response is unrehearsed.
  • ⚠No debrief or action items — the game day ran but the stale runbook it found stays stale, so the next real outage hits the same gap.
  • ⚠Only run once — game days need to be quarterly (at least) so new on-call engineers get the IC muscle before a real SEV1.

Best Practices

  • ✓Pick a plausible SEV1 scenario (DB failure, region down, bad deploy) and run the full incident response in staging — IC, war room, comms, rollback.
  • ✓Debrief immediately and track action items (stale runbook, missing alert, IC gap) like postmortem items — the game day's value is the fixes it produces.
  • ✓Run quarterly for Tier 1 services and whenever a new on-call joins; the first time someone is IC should be a game day, not a real SEV1.
  • ✓A game day that finds 3 broken runbooks is a success; one that finds nothing is a worry — you're either not injecting hard enough or not looking closely.