Skip to content
Back to Operations & SRE

Capacity Headroom

Maintain measured capacity headroom so traffic spikes don't cause saturation outages.

Why

A service running at 95% CPU is one viral tweet away from an outage — there's no headroom for a 20% traffic spike, a retry storm, or a deploy that adds 5% load. Capacity headroom is the buffer between your current utilization and the point where the service degrades, and maintaining it (typically 30-50% headroom on critical resources) is the cheapest reliability investment you can make. The trap is that headroom feels like wasted money — "we're paying for servers we don't need" — but the cost of 30% over-provisioning is a fraction of the cost of one saturation outage that loses revenue and customer trust. Teams that track headroom and alert when it drops below 30% catch capacity issues during normal growth, not during a spike that becomes an outage.

When

For every stateful resource (DB, cache, queue) and every autoscaled service's baseline. Review before expected spikes (launch, marketing, seasonal).

How

1. Identify the saturation point per resource
   - DB: connection pool exhaustion, CPU >80%, disk IOPS limit
   - Cache: eviction rate >1%, memory >90%
   - Queue: depth growing unboundedly, consumer lag increasing
   - Service: p99 latency rising sharply at a specific QPS (the "knee")

2. Measure current utilization vs the saturation point
   - headroom = (saturation_point - current_utilization) / saturation_point
   - Track on a dashboard per resource; alert when headroom <30%

3. Set a headroom target per resource class
   - Stateful (DB, cache): 30-50% headroom (hard to scale quickly, need buffer)
   - Stateless autoscaled: 20% headroom (autoscaler handles spikes, just need baseline buffer)
   - Single-instance (no HA): 50%+ headroom (no failover, need margin for degradation)

4. Alert on headroom erosion, not just absolute utilization
   - "DB CPU >70% for 1h" → ticket (headroom <30%, plan capacity)
   - "DB CPU >85%" → page (headroom <15%, imminent saturation risk)
   - Track the trend, not just the snapshot — 70% rising 1%/week is urgent

5. Plan capacity from growth rate + spike buffer
   - current × (1 + weekly_growth) × (1 + spike_buffer) = capacity needed in N weeks
   - Order hardware/resize before you need it, not when the alert fires

Example

# Capacity headroom dashboard (per resource)
# DB (Postgres):
db_cpu = 68%  # saturation at 85% → headroom = (85-68)/85 = 20%  ⚠ <30%, ticket
db_connections = 180/200  # 90% → headroom = 10%  🔴 page
db_disk_iops = 12000/15000  # 80% → headroom = 20%  ⚠

# Cache (Redis):
cache_memory = 78%  # eviction at 90% → headroom = 13%  ⚠
cache_eviction_rate = 0.3%  # <1% ok

# Alert rules:
- alert: DbHeadroomLow
  expr: (85 - avg(db_cpu_pct)) / 85 < 0.30  # headroom <30%
  for: 1h
  labels: { severity: ticket, team: platform }
  annotations: { runbook: "https://rb/db-capacity-plan" }

# Capacity plan (growth-based):
#   current_qps = 500, weekly_growth = 2%, spike_buffer = 50% (Black Friday)
#   qps_in_8_weeks = 500 × 1.02^8 × 1.5 = 500 × 1.17 × 1.5 = 878
#   → need capacity for 878 QPS by week 8; order the upgrade now

Common Problems

  • ⚠Running at 90%+ to "save money" — the savings are a fraction of one saturation outage's cost; headroom is the cheapest insurance you can buy.
  • ⚠Alerting on absolute utilization (CPU >80%) instead of headroom — 80% of a 100-conn pool is fine, 80% of a 200-conn pool with a 250 limit is not; track the margin.
  • ⚠No growth-rate tracking — you hit 70% and order hardware that takes 4 weeks to arrive, by which time you're at 88% and saturating.
  • ⚠Autoscaler hides the baseline headroom problem — the autoscaler keeps adding instances but the DB (which can't autoscale) saturates; track every resource, not just the easy ones.

Best Practices

  • ✓Maintain 30-50% headroom on stateful resources (DB, cache) that can't autoscale quickly; 20% on autoscaled stateless services is usually enough.
  • ✓Alert on headroom (margin to saturation), not absolute utilization — the same percentage means different margins for different pool sizes.
  • ✓Track growth rate and order capacity N weeks before you need it, accounting for procurement lead time — don't order when the alert fires.
  • ✓Before expected spikes (launch, Black Friday), pre-scale and load-test to confirm headroom; don't rely on the autoscaler to save you during a spike.