Log Aggregation

Centralize logs from all services in a single aggregator with retention, search, and alerting.

Why

Logs on the local disk are invisible during an incident — the pod that crashed is gone, the auto-scaler terminated the instance, and the on-call can't SSH into a box that doesn't exist. A log aggregator (Loki, Datadog, ELK, CloudWatch) ingests logs from all services in real time, so the logs survive the pod death and are queryable from a single pane. This is the difference between 'the logs are gone' and 'query the last 5m of checkout ERROR logs in 50ms'. The aggregator also provides retention (30-90 days for incident forensics), search (structured queries, not grep), and alerting (alert on log patterns, not just metrics). Without aggregation, every incident starts with 'where are the logs?' — with it, the logs are just there, queryable, and the on-call skips 10 minutes of hunting and goes straight to diagnosis.

When

From day one. A service without log aggregation is a service you can't debug in prod.

How

1. Choose an aggregator and ship logs from all services
   - Loki (Grafana stack, cheap, good for structured logs), Datadog (paid, full-featured),
     ELK (self-hosted, powerful but ops-heavy), CloudWatch (AWS-native)
   - Ship via agent (Promtail, Fluentd, Datadog agent) or directly from the app

2. Enforce a retention policy
   - 7-30 days hot (queryable, fast), 90 days warm (queryable, slower), 1-7 years cold (compliance)
   - Hot retention is expensive; don't keep 90 days hot unless you query that far often

3. Make logs queryable by the fields that matter
   - Index or label by: service, level, trace_id, route, status_code
   - Don't index everything (expensive); index the fields you query by

4. Build saved queries for common incident patterns
   - "checkout errors last 5m": {service="checkout", level="ERROR"} | last 5m
   - "slow requests": {latency_ms > 1000} | last 1h
   - The on-call runs a saved query, not a custom one, during an incident

5. Alert on log patterns, not just metrics
   - "ERROR rate > 1/s for 5m" → page
   - "specific error message appears" → page (e.g., "DB connection refused")
   - Logs catch failures that metrics miss (e.g., a rare error that doesn't move the error-rate metric)

Example

# Log aggregation architecture:
App → logger (JSON) → Promtail → Loki → Grafana (query/dashboard/alert)
                       ↓
                       Datadog agent → Datadog (if using both)

# Retention policy (Loki):
retention:
  hot:  30 days   # fast queries, expensive storage
  warm: 90 days   # slower queries, cheaper storage
  cold: 7 years   # compliance, rarely queried

# Saved queries (Grafana):
- name: "checkout-errors-5m"
  query: '{service="checkout", level="ERROR"} |= "" | last 5m'
- name: "slow-requests-1h"
  query: '{latency_ms > 1000} | last 1h'
- name: "db-connection-refused"
  query: '{msg=~"DB connection refused"} | last 15m'

# Alert on log pattern (not just metric):
- alert: CheckoutDbConnectionRefused
  expr: count_over_time({service="checkout", msg=~"DB connection refused"}[5m]) > 0
  annotations:
    runbook_url: "https://rb/checkout-db-failover"
# This catches the specific failure even if the error-rate metric hasn't breached

Common Problems

⚠Logs on local disk only — the pod crashes, the logs are gone, and the incident starts with 'where are the logs?'
⚠No retention policy — logs expire in 3 days, so the postmortem a week later can't see what happened.
⚠Indexing every field — storage costs explode; index only the fields you query by (service, level, trace_id).
⚠Only metric alerts, no log alerts — a rare error (DB connection refused) doesn't move the error-rate metric but is still a failure; alert on the log pattern too.

Best Practices

✓Ship logs from all services to a central aggregator (Loki, Datadog, ELK) in real time; logs survive pod death and are queryable from one pane.
✓Set a tiered retention policy (hot 30d, warm 90d, cold 7y for compliance); don't keep everything hot — it's expensive.
✓Index only the fields you query by (service, level, trace_id, route); build saved queries for common incident patterns so the on-call runs, not writes, a query.
✓Alert on log patterns (specific error messages, error-rate by service) in addition to metrics; logs catch failures that metrics miss.