Toil Reduction
Identify and eliminate repetitive, manual, automatable operational work that drains engineering time.
Why
Toil is the repetitive, manual, automatable work that grows to consume 50%+ of an SRE's week if left unchecked — restarting pods, copying log files, running the same SQL to generate a report, manually failover a queue. Every hour of toil is an hour not spent on engineering work that reduces future toil, so toil compounds: the more you have, the less time you have to eliminate it, so it grows. The SRE prescription is a toil budget (cap toil at <50% of time) and a standing toil-reduction queue: every quarter, pick the top toil source and automate or eliminate it. Teams that do this report toil dropping from 60% to 20% of their time within a year, freeing the rest for reliability projects that prevent outages.
When
When operational work consumes >50% of an engineer's week, or when the same manual task has been done 3+ times. Review quarterly.
How
1. Measure toil for 2 weeks - Log every task: what, time spent, manual/repetitive/automatable - Toil = manual + repetitive + automatable + no enduring value - Total toil hours / total hours = toil percentage 2. Build the toil queue - Rank by (time spent × frequency) — biggest drain first - Tag each: "automate" / "eliminate" / "document and accept" - The queue is a backlog item, reviewed in sprint planning 3. Pick the top item and eliminate it - Automate: write a script/runbook that does it; trigger from an alert or schedule - Eliminate: is the task necessary? Often a process exists long after its reason - Document + accept: if truly necessary and not automatable, make it 5-min not 30-min 4. Cap toil at <50% of time - If toil >50%, toil reduction is the sprint's top priority, not feature work - This is the SRE "toil budget" — a hard cap, not an aspiration 5. Prevent new toil - Every new manual procedure needs a "how do we automate this" plan - Code review: if a change adds a manual step, require the automation too
Example
# Toil log (2-week sample): # restart-pod-orders 15m × 12 times = 3h → automate (runbook-trigger) # generate-weekly-report 45m × 1 time = 45m → automate (cron + SQL) # copy-logs-for-debug 10m × 8 times = 80m → automate (dashboard link) # manual-failover-queue 30m × 2 times = 60m → automate (auto-failover) # Total toil: 5h 25m / 80h = 6.8% (healthy, but restart-pod is the top drain) # Automate restart-pod-orders (the top item): # Before: on-call SSHes to the box, runs kubectl delete pod, waits, confirms # After: alert auto-triggers runbook that deletes the pod and verifies recovery # - runbook: https://rb/restart-orders-pod (auto-executable, see runbook-trigger) # - Saves 3h/week of toil, and reduces MTTR by 10m (no human login time) # Toil budget policy: # if toil_pct > 50: sprint top priority = toil reduction (no features) # review the toil log every quarter, re-rank, eliminate the top item
Common Problems
- ⚠Not measuring toil — without a 2-week log, toil is a vibe, and engineers can't justify time to automate over feature work.
- ⚠Automating toil but not maintaining the automation — the script rots and breaks, reverting to manual and adding 'fix the script' toil on top.
- ⚠Treating toil reduction as a one-time project instead of a standing queue — new toil accretes with every feature, so the queue must be permanent.
- ⚠No toil budget cap — toil grows to fill available time because feature work is always more visible than the toil it creates.
Best Practices
- ✓Log toil for 2 weeks to quantify it — without numbers, toil reduction loses to feature work in every sprint planning conversation.
- ✓Keep a permanent toil-reduction backlog ranked by time×frequency; eliminate the top item every quarter, not whenever you have spare time.
- ✓Cap toil at <50% of engineering time as a hard budget; when exceeded, toil reduction is the sprint's top priority, not features.
- ✓Require every new manual procedure to ship with an automation plan — code review rejects changes that add permanent manual steps.