Operations &amp; SRE

Metrics Design

Choose the right metrics (RED, USE, RUM) and instrument them consistently across services.

Dashboards

Build dashboards that on-call engineers can read at 3am — fast, obvious, and action-oriented.

Synthetic Monitoring

Run scripted transactions against production from external probes to catch outages before users do.

Real User Monitoring

Collect performance and errors from real user browsers to see what users actually experience.

Uptime Checks

Lightweight TCP/HTTP pings from multiple regions to detect reachability outages with minimal cost.

APM Setup

Install Application Performance Monitoring to get distributed traces, service maps, and per-request profiling.

Alert Rules

Define alert rules that fire on user-visible symptoms, not internal noise, with clear thresholds and ownership.

Escalation Policy

Define who gets paged, in what order, and when to escalate if the first responder doesn't acknowledge.

On-Call Rotation

Schedule on-call shifts fairly across the team with follow-the-sun, handoffs, and compensation policies.

Alert Fatigue

Reduce alert volume so each page is meaningful — pruning, aggregation, and noise budgets.

Runbook Trigger

Link every alert to a runbook and auto-trigger executable runbooks where the first step is deterministic.

Severity Mapping

Map every incident to a severity (SEV1-SEV4) based on user and business impact, not technical cause.

SLO Definition

Define Service Level Objectives that tie technical reliability to user happiness with measurable targets.

Error Budget

Track the error budget as a shared scoreboard that governs deploy velocity and reliability investment.

Toil Reduction

Identify and eliminate repetitive, manual, automatable operational work that drains engineering time.

Blameless Postmortem

Run blameless postmortems that find systemic causes and produce action items, not scapegoats.

Reliability Target

Set reliability targets per service tier so investment matches the service's criticality to the business.

Capacity Headroom

Maintain measured capacity headroom so traffic spikes don't cause saturation outages.

Incident Severity

Declare severity within 5 minutes of detection to set response urgency and escalation.

War Room

Open a dedicated incident channel and bridge for SEV1 to concentrate coordination and reduce cross-team friction.

Comms Protocol

Define who communicates what to whom during an incident — internal, stakeholders, customers.

Rollback Playbook

Make rollback a one-command, well-rehearsed action so deploys are reversible in under 5 minutes.

Customer Comms

Communicate with customers during outages via a status page, email, and in-app banners with honesty and cadence.

Post-Incident Review

Turn every incident into systemic improvements via a blameless postmortem with tracked action items.

Incident Commander

Designate an Incident Commander who coordinates, decides, and communicates — not the person debugging.

Communications Protocol

Keep customers and stakeholders informed during incidents with a predictable cadence and status page.

War Room

Convene a focused war room for SEV1 with the right people, a clear scope, and a time limit.

Postmortem Process

Run a structured postmortem within 5 business days that produces a timeline, root cause, and tracked action items.

Game Day

Run game days that inject failures into staging to practice incident response before a real outage.

MTTR Tracking

Track Mean Time To Resolve as a leading indicator of incident response quality, alongside MTTA and MTTF.

Runbook Structure

Structure runbooks with a fixed template: symptom, diagnosis, steps, verification, escalation.

Runbook Ownership

Assign every runbook an owner and a team; review quarterly so runbooks don't rot.

Executable Runbook

Make runbooks executable scripts, not just docs — the on-call runs the runbook, they don't read it.

Runbook Testing

Test runbooks in staging regularly so the first execution during a real incident isn't the first test.

Runbook Discovery

Make runbooks discoverable from the alert, the dashboard, and the service catalog — not buried in a wiki.

Runbook Versioning

Version runbooks in git alongside the code so they evolve with the system and are reviewable.

Structured Logging

Emit logs as JSON with fields (timestamp, level, service, trace_id, message), not unstructured text.

Log Aggregation

Centralize logs from all services in a single aggregator with retention, search, and alerting.

Log Sampling

Sample high-volume non-error logs to control cost while keeping 100% of errors and warnings.

Log Alerting

Alert on log patterns — specific errors, error rates, and absence-of-expected-logs — not just metrics.

Log Retention

Set tiered retention (hot/warm/cold) that balances query speed, cost, and compliance needs.

Log Correlation

Correlate logs with traces and metrics using trace_id to see the full request path during incidents.

Headroom Targets

Maintain 30-50% headroom on critical resources so traffic spikes don't become outages.

Demand Forecasting

Forecast demand from historical trends, seasonality, and business events to pre-provision capacity.

Auto-Scaling

Configure auto-scaling that responds fast to traffic changes with appropriate metrics and cooldowns.

Load Testing

Load test to capacity limits before production so you know the ceiling and can plan headroom.

Capacity Alerting

Alert on capacity exhaustion (headroom, saturation, growth rate) before it becomes an outage.