Auto-Scaling

Configure auto-scaling that responds fast to traffic changes with appropriate metrics and cooldowns.

Why

Manual scaling doesn't scale — the on-call can't watch 50 services and scale each one as traffic shifts. Auto-scaling (K8s HPA, KEDA, cloud auto-scaler) watches a metric (CPU, queue depth, custom) and adjusts replicas automatically, so a traffic spike scales the service in 30-60 seconds without human intervention. The key is configuring it right: scale on the right metric (queue depth for workers, CPU for web services, RPS for latency-sensitive), scale fast enough (30s stabilization for scale-up, 5m for scale-down to avoid flapping), and set sane bounds (min replicas to handle baseline, max to cap cost). Misconfigured auto-scaling is worse than none — a HPA that scales on average CPU across 1 pod does nothing when that pod is at 100% (average is 100%, target is 70%, it scales to 2, but the new pod takes 60s to start and the original is still at 100%). Get the config right, test it with load tests, and let it handle the routine scaling while humans handle the exceptions.

When

For all services that see variable traffic. Configure auto-scaling as the baseline; use forecasting for predictable spikes.

How

1. Choose the right scaling metric per service type
   - Web services: CPU (70% target) or RPS per pod (latency-sensitive)
   - Workers: queue depth (scale when backlog > N)
   - Batch: job completion rate or queue age
   - Don't scale web services on queue depth or workers on CPU — wrong metric

2. Set fast scale-up, slow scale-down
   - Scale up: 0-30s stabilization (react fast to spikes)
   - Scale down: 5m stabilization (don't flap on traffic oscillation)
   - K8s: behavior.scaleUp.stabilizationWindowSeconds: 0, scaleDown: 300

3. Set min and max replicas
   - Min: enough to handle baseline traffic + 1 (for pod restarts without downtime)
   - Max: cap to prevent runaway cost (e.g., 50 pods max); alert if you hit max
   - Max should be high enough for spikes but low enough to not bankrupt you

4. Test auto-scaling with load tests
   - Ramp traffic 2x, 5x, 10x → does it scale? How fast? Does latency stay in SLO?
   - Test scale-down: drop traffic → does it scale down without dropping in-flight requests?

5. Monitor and tune
   - Track scaling events: how often, how fast, did it keep up?
   - If it's always at max → raise max or fix the bottleneck (DB, not pods)
   - If it flaps → increase scale-down stabilization

Example

# K8s HPA for checkout (web service, CPU-based):
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: checkout
spec:
  scaleTargetRef:
    kind: Deployment
    name: checkout
  minReplicas: 3          # baseline + 1 (handles restarts)
  maxReplicas: 50         # cap; alert if hit
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 70   # 30% headroom
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 0    # scale up immediately
      policies:
        - type: Percent
          value: 100                   # can double per minute
          periodSeconds: 60
    scaleDown:
      stabilizationWindowSeconds: 300  # 5m before scaling down
      policies:
        - type: Percent
          value: 10                    # slow scale-down (10% per minute)
          periodSeconds: 60

# KEDA for order-worker (queue-based):
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: order-worker
spec:
  scaleTargetRef:
    name: order-worker
  minReplicaCount: 1
  maxReplicaCount: 20
  triggers:
    - type: aws-sqs
      metadata:
        queueName: orders
        queueLength: "10"    # 1 pod per 10 messages in queue

# Alert if at max (can't scale further):
- alert: CheckoutAtMaxReplicas
  expr: kube_deployment_status_replicas{deployment="checkout"} == 50
  for: 5m
  labels: { severity: page }
  annotations:
    summary: "checkout at max replicas (50) — traffic exceeds capacity, raise max or fix bottleneck"

Common Problems

⚠Scaling on the wrong metric — CPU for a queue-processing worker (should be queue depth), or queue depth for a web service (should be CPU/RPS).
⚠Scale-up too slow — 5m stabilization means the spike is over before the pods are ready; use 0-30s for scale-up.
⚠No max replicas — runaway scaling bankrupts the team; set a max and alert when you hit it (means you need to fix the bottleneck, not add more pods).
⚠Scale-down flapping — 0s stabilization means pods scale down and back up every 30s as traffic oscillates; use 5m for scale-down.

Best Practices

✓Scale on the right metric per service type: CPU/RPS for web, queue depth for workers, job rate for batch.
✓Set fast scale-up (0-30s stabilization) and slow scale-down (5m) to react fast to spikes without flapping on oscillation.
✓Set min (baseline + 1) and max replicas; alert when you hit max — it means traffic exceeds capacity and you need to fix the bottleneck, not add pods.
✓Test auto-scaling with load tests (2x, 5x, 10x) and monitor scaling events; tune stabilization and policies based on actual behavior.