Anomaly Detection

Why

When

How

1. Collect 4-8 weeks of clean historical data for the metric at the granularity you will alert on (hourly or daily)
2. Choose a method: start with simple (mean +/- 3*std for Gaussian, or median +/- 3*MAD for robust), graduate to seasonal (STL decomposition, Prophet) only if the simple method has > 5% false positives
3. Account for seasonality: day-of-week (weekend traffic is lower), hour-of-day (3am is always low), holidays —subtract the expected seasonal component before testing for anomaly
4. Set the alert threshold at the point where false-positive rate is acceptable (start at 1 false alert/week, tighten as trust grows)
5. Add a dead-man-switch: if no data arrives for 2 intervals, alert —a broken pipeline reports no anomaly because there is no number to test
6. Route alerts to the on-call channel with the metric name, current value, expected range, and a deep link to the dashboard —not just "metric anomalous"
7. Run a 2-week shadow period where alerts are logged but not sent —measure false-positive rate before turning on paging
8. Review and re-tune quarterly: as the product grows, variance changes, and a threshold that was right at 1k users/day is wrong at 10k

Example

# Anomaly: deploy success rate dropped 2026-09-21 14:00 UTC
Metric: deploy_success_rate (hourly, last 6 weeks baseline)
Current: 0.71  Expected: [0.92, 0.99]  Z-score: -4.2

Alert routed to: #oncall-platform
Runbook: https://wiki/runbooks/deploy-success-drop

Investigation:
  14:03  on-call acked alert
  14:05  checked deploy log: 7 consecutive failures on eu-west region
  14:08  root cause: CDN config push broke asset resolution in eu-west only
  14:12  reverted CDN config
  14:15  success rate back to 0.97
  14:20  postmortem doc opened

Without anomaly detection: would have been found in next morning's
standup (18h later), affecting ~400 deploys in eu-west.

Config (tuned after 2-week shadow):
  method: robust (median +/- 3*MAD)
  seasonality: day-of-week, hour-of-day
  threshold: 3.0 (z-score)
  min_samples: 336 (2 weeks hourly)
  false-positive rate: 0.8/week (acceptable)

Common Problems

• ⚠Alerting on raw value instead of deviation from expected —a 3am metric is always low, so a fixed threshold fires every night at 3am and the team learns to ignore all anomaly alerts, including the real one at 2pm.
• ⚠No dead-man-switch on the data pipeline —the most catastrophic anomaly (no data at all) is the one that never fires an alert because there is no number to test, so a silent dashboard is treated as a good day.
• ⚠Tuning the threshold too tight to avoid false positives —the team sets the threshold at 5 sigma to 'stop the noise', and the threshold becomes so high that the only anomalies that fire are ones a human would have noticed anyway, defeating the purpose.
• ⚠Alerting on every metric instead of the 10 that matter —200 anomaly alerts means 190 are ignored, and the 10 critical ones are buried in noise; the alert budget is finite and should be spent on metrics whose anomaly implies user-visible damage.

Best Practices

• ✓Start with robust simple methods (median +/- 3*MAD) before reaching for ML —the simple method catches 90% of anomalies, is explainable to a non-technical stakeholder, and fails in predictable ways; ML fails in mysterious ways that erode trust.
• ✓Always subtract seasonality before testing for anomaly —a Tuesday drop from 1000 to 700 is an anomaly; a Sunday drop from 1000 to 700 is just the weekend, and conflating them destroys alert credibility.
• ✓Include a deep link to the dashboard and the runbook in every alert —the on-call's first 5 minutes are spent deciding 'is this real', and a link to context turns that into 30 seconds, which over a year saves hundreds of engineer-hours.
• ✓Re-tune thresholds after any product change that affects volume (launch, pricing change, new market) —the baseline distribution shifts, and a threshold tuned to the old distribution will either fire constantly or never fire at all.