Headroom Targets

Maintain 30-50% headroom on critical resources so traffic spikes don't become outages.

Why

A service running at 95% CPU has 5% headroom — a 10% traffic spike (Black Friday, a viral post, a downstream retry storm) pushes it to 105%, latency spikes as the scheduler thrashes, and the service falls over. A 30-50% headroom target (70% CPU utilization at peak) absorbs normal spikes without auto-scaling lag, and gives auto-scaling time to kick in for larger ones. This is cheap insurance: the cost of 30% over-provisioning is 30% of your compute bill, but the cost of an outage during peak (lost revenue, customer churn, SLA penalties) is 10-100x that. The trap is running at 90% to 'save money' — you save 10% on compute and pay 100% on the next outage. Set headroom per resource (CPU, memory, DB connections, queue depth) and alert when utilization exceeds the target, so you scale before you're out.

When

Always. Set headroom targets per critical resource and alert when utilization exceeds them.

How

1. Identify critical resources and their limits
   - CPU, memory, disk I/O, network (per pod/instance)
   - DB connections (pool limit), DB CPU, DB storage
   - Queue depth (SQS, Kafka), cache hit rate, cache memory

2. Set a headroom target per resource
   - CPU: 70% at peak (30% headroom) for normal, 50% (50% headroom) for peak events
   - Memory: 80% at peak (20% headroom; OOM is harder than CPU throttle)
   - DB connections: 70% of pool (30% headroom for burst)
   - Queue depth: <50% of consumer capacity (50% headroom for burst)

3. Alert when utilization exceeds the target
   - CPU >70% for 5m → warn (scale or investigate)
   - CPU >85% for 2m → page (outage imminent)
   - DB connections >80% → page (pool exhaustion → request failures)

4. Track utilization at peak, not average
   - Average CPU 40% looks fine, but peak is 92% → you're out of headroom at peak
   - Alert on p95/p99 utilization, not mean

5. Re-evaluate headroom after traffic changes
   - New customer doubling traffic → re-check headroom before the traffic hits
   - Seasonal peak (holiday shopping) → temporarily raise headroom target to 50%

Example

# Headroom targets (per resource):
resource_targets:
  cpu:        warn: 70%, page: 85%   # 30% headroom at peak
  memory:     warn: 80%, page: 90%   # 20% headroom (OOM is hard)
  db_conns:   warn: 70%, page: 80%   # 30% headroom for burst
  queue_depth: warn: 50%, page: 75%  # 50% headroom for burst

# Alerts (Prometheus):
- alert: CheckoutCPUHigh
  expr: quantile_over_time(0.95, rate(cpu_usage_seconds[5m])) > 0.70
  for: 5m
  labels: { severity: warn }
  annotations:
    summary: "checkout CPU p95 >70% — headroom shrinking, consider scaling"

- alert: CheckoutCPUCritical
  expr: quantile_over_time(0.95, rate(cpu_usage_seconds[5m])) > 0.85
  for: 2m
  labels: { severity: page }
  annotations:
    summary: "checkout CPU p95 >85% — outage imminent, scale NOW"

# Dashboard (track peak, not average):
#   checkout CPU:  mean 40%, p95 72%, p99 91%  ← p99 exceeds target!
#   → Action: scale checkout +2 pods (brings p99 to ~75%)
#   → Root cause: a slow downstream causes CPU spikes on retries

Common Problems

⚠Tracking average, not peak — average CPU 40% looks fine but peak is 92%; alert on p95/p99, not mean.
⚠No headroom on DB connections — the pool hits 100% during a burst and all requests fail; keep 30% headroom on the pool.
⚠Running at 90% to save money — you save 10% on compute and pay 100% on the next outage; headroom is cheap insurance.
⚠Static headroom that doesn't adjust for events — 30% is fine for normal traffic but Black Friday needs 50%; raise the target for known peaks.

Best Practices

✓Set 30-50% headroom on critical resources (CPU, memory, DB connections, queue depth) and alert when p95 utilization exceeds the target.
✓Alert on peak (p95/p99) utilization, not average; mean hides the spikes that cause outages.
✓Treat headroom as cheap insurance: 30% over-provisioning costs 30% of compute, but an outage during peak costs 10-100x that in lost revenue.
✓Raise headroom targets for known peaks (Black Friday, product launch) before the traffic hits; re-evaluate after traffic changes.