Load Testing

Load test to capacity limits before production so you know the ceiling and can plan headroom.

Why

You don't know your service's capacity until you break it — and finding out in production is the worst way. A load test ramps traffic until the service fails (latency > SLO, error rate > 0, OOM), and tells you the ceiling: 'checkout handles 5,000 RPS at p99 <500ms, fails at 6,000'. This is the data you need to plan headroom: if peak traffic is 1,000 RPS and the ceiling is 5,000, you have 5x headroom; if peak is 4,000, you have 1.25x and need to scale. Load testing before launch (new service, major feature, capacity change) prevents the 'we launched and it immediately fell over' failure mode. It also validates auto-scaling: ramp to 10x and confirm the HPA scales and latency stays in SLO. Teams that load test before every launch report 80% fewer capacity incidents, because they know the ceiling and provision to stay under it.

When

Before launch (new service, major feature), before capacity changes, and quarterly for Tier 1 services.

How

1. Define the load test profile
   - Production-like traffic mix (read/write ratio, endpoint distribution)
   - Ramp pattern: gradual (10% → 100% → 200%) or spike (0 → 10x in 1m)
   - Duration: long enough to hit steady state (10-30m) and find memory leaks

2. Test in a prod-like environment
   - Staging with prod-like capacity, data volume, and downstream deps
   - Testing in a 1-pod staging tells you nothing about prod behavior
   - Use prod-like data (anonymized) so caches and DB behave realistically

3. Ramp until the service fails
   - Increase RPS until: latency > SLO, error rate > 0, or resource exhaustion (OOM, CPU 100%)
   - Record the ceiling: "5,000 RPS at p99 <500ms; 6,000 RPS → p99 2s, errors 1%"
   - The failure mode tells you the bottleneck (CPU, DB, memory, downstream)

4. Validate auto-scaling during the test
   - Ramp to 2x expected peak → does HPA scale? Does latency stay in SLO?
   - If HPA doesn't keep up → tune stabilization, max replicas, or scale-up policy

5. Document the ceiling and plan headroom
   - "Ceiling: 5,000 RPS. Peak traffic: 1,000 RPS. Headroom: 5x. Action: none."
   - "Ceiling: 5,000 RPS. Peak traffic: 4,000 RPS. Headroom: 1.25x. Action: scale +50%."
   - Re-test after changes that affect capacity (new code, new infra, traffic growth)

Example

# Load test (k6, checkout service):
import http from 'k6/http';
import { check, sleep } from 'k6';

export const options = {
  stages: [
    { duration: '2m', target: 1000 },   # ramp to 1000 RPS
    { duration: '5m', target: 1000 },   # hold 1000
    { duration: '2m', target: 3000 },   # ramp to 3000
    { duration: '5m', target: 3000 },   # hold 3000
    { duration: '2m', target: 5000 },   # ramp to 5000
    { duration: '5m', target: 5000 },   # hold 5000
    { duration: '2m', target: 6000 },   # ramp to 6000 (expected failure)
    { duration: '5m', target: 6000 },   # hold 6000
  ],
  thresholds: {
    http_req_duration: ['p(99)<500'],   # SLO: p99 < 500ms
    http_req_failed: ['rate<0.01'],     # SLO: error rate < 1%
  },
};

export default function () {
  const res = http.post('https://staging.yourcompany.com/checkout', payload);
  check(res, { 'status 200': (r) => r.status === 200 });
  sleep(0.1);
}

# Results:
#   1000 RPS: p99 120ms, errors 0%   ✅
#   3000 RPS: p99 280ms, errors 0%   ✅
#   5000 RPS: p99 450ms, errors 0.1% ✅ (at SLO limit)
#   6000 RPS: p99 2100ms, errors 5%  ❌ (broke SLO)
# Ceiling: 5,000 RPS. Bottleneck: DB connection pool (exhausted at 5500).

# Capacity plan:
#   Peak traffic: 1,000 RPS. Ceiling: 5,000 RPS. Headroom: 5x. ✅ No action.
#   If peak grows to 4,000 RPS → headroom 1.25x → scale checkout +50% (raise DB pool too).

Common Problems

⚠Testing in a 1-pod staging — tells you nothing about prod behavior; use prod-like capacity, data, and deps.
⚠Ramping to expected peak and stopping — you know you can handle peak but not the ceiling; ramp until failure to find the limit.
⚠No auto-scaling validation — the service handles 5k RPS with 50 pods, but does the HPA actually scale to 50? Test it.
⚠One-time test, never re-run — the ceiling drops as the code grows; re-test quarterly and after capacity-affecting changes.

Best Practices

✓Load test in a prod-like environment (capacity, data, deps) ramping until the service fails; the ceiling is the data you need to plan headroom.
✓Validate auto-scaling during the test: ramp to 2x expected peak and confirm the HPA scales and latency stays in SLO.
✓Document the ceiling and plan headroom: if peak/ceiling < 2x, scale up; re-test after changes that affect capacity.
✓Re-test quarterly for Tier 1 services and after capacity-affecting changes (new code, new infra, traffic growth); the ceiling drops as the code grows.