Capacity Alerting

Alert on capacity exhaustion (headroom, saturation, growth rate) before it becomes an outage.

Why

An outage alert ('service down') fires after the damage — customers are already failing. A capacity alert ('headroom < 10%', 'DB pool 80% full', 'disk 90%') fires before the damage, giving you minutes to hours to scale or fix before customers are affected. This is the shift from reactive to proactive ops: alert on the leading indicators (utilization, saturation, growth rate) instead of the lagging ones (errors, latency). The three capacity signals are: utilization (CPU 70%, disk 80%), saturation (queue depth growing, DB pool waiting), and growth rate (traffic growing 20% MoM, will hit ceiling in 2 weeks). Alert on all three, with enough lead time to act — 'disk 90%' gives you hours, 'disk 99%' gives you minutes, and 'disk full' gives you an outage. Teams that alert on capacity report 70% fewer customer-impacting incidents, because they fix the capacity issue before it breaks anything.

When

Always. Set capacity alerts on all critical resources with thresholds that give lead time to act.

How

1. Alert on utilization (the basic signal)
   - CPU >70% for 5m → warn; >85% for 2m → page
   - Memory >80% → warn; >90% → page (OOM is imminent)
   - Disk >80% → warn; >90% → page (disk full = service down)

2. Alert on saturation (the growing signal)
   - Queue depth growing (depth now > 2x depth 5m ago) → warn
   - DB pool waiting (wait_time > 100ms) → warn; >500ms → page
   - Thread pool exhausted (rejected tasks > 0) → page

3. Alert on growth rate (the predictive signal)
   - Traffic growing >20% WoW → warn (will hit ceiling if trend continues)
   - Disk usage growing >5%/day → warn (will fill in N days)
   - DB size growing >10%/month → warn (will hit storage limit)

4. Set thresholds with lead time
   - Disk 80% → 2 days to full at 10%/day → warn (time to act)
   - Disk 90% → 1 day to full → page (act now)
   - Don't set the threshold at 99% — that's 1 hour to full, not enough lead time

5. Route capacity alerts to the right responder
   - Disk → platform team (add storage or clean up)
   - DB pool → DB team (raise pool or tune queries)
   - Traffic growth → service team (scale or investigate the growth cause)

Example

# Capacity alerts (Prometheus):
groups:
  - name: capacity
    rules:
      # 1. Utilization (basic)
      - alert: CheckoutCPUHigh
        expr: avg(rate(cpu_usage[5m])) > 0.70
        for: 5m
        labels: { severity: warn, team: payments }
      - alert: CheckoutCPUCritical
        expr: avg(rate(cpu_usage[5m])) > 0.85
        for: 2m
        labels: { severity: page, team: payments }

      # 2. Saturation (growing)
      - alert: CheckoutQueueGrowing
        expr: queue_depth > 2 * queue_depth offset 5m
        labels: { severity: warn, team: payments }
      - alert: DbPoolWaiting
        expr: db_pool_wait_time_ms > 500
        for: 1m
        labels: { severity: page, team: db-platform }

      # 3. Growth rate (predictive)
      - alert: DiskFillingFast
        expr: deriv(disk_usage[1d]) > 0.05  # >5%/day
        labels: { severity: warn, team: platform }
        annotations:
          summary: "Disk filling at >5%/day — will fill in {{ $value }} days"
      - alert: TrafficGrowingFast
        expr: rate(http_requests[1w]) > 1.2  # >20% WoW
        labels: { severity: warn, team: payments }
        annotations:
          summary: "Traffic growing >20% WoW — check capacity vs ceiling"

# Lead time:
#   Disk 80% at 5%/day → 4 days to full → warn (4 days lead time)
#   Disk 90% at 5%/day → 2 days to full → page (2 days lead time)
#   Disk 95% at 5%/day → 1 day to full → page (1 day lead time, act NOW)

Common Problems

⚠Only outage alerts, no capacity alerts — you find out when customers fail; alert on utilization/saturation/growth to get lead time.
⚠Threshold at 99% — 1 hour to full, not enough lead time to act; set at 80% (warn) and 90% (page) for days of lead time.
⚠No growth-rate alerts — traffic grows 20% MoM and hits the ceiling in 2 weeks with no warning; alert on the trend, not just the current value.
⚠All capacity alerts to one team — disk alerts to the app team who can't add storage; route by resource to the team that can fix it.

Best Practices

✓Alert on all three capacity signals: utilization (CPU/disk %), saturation (queue/pool waiting), and growth rate (traffic/disk growing).
✓Set thresholds with lead time: disk 80% warn (days to full), 90% page (act now); don't set at 99% — that's an outage, not a capacity alert.
✓Alert on growth rate (deriv over 1d/1w) to predict ceiling hits weeks before they happen; the trend is the early warning.
✓Route capacity alerts by resource to the team that can fix it (disk → platform, DB pool → DB team, traffic → service team).