Observability Tooling
Pick the metrics/tracing/logging stack that lets you debug production in minutes, not hours.
Why
When production breaks, the observability stack is the only thing between you and a blind debug. The right tools (metrics for 'what', traces for 'where', logs for 'why', structured + correlated) let you find the cause in minutes. The wrong tools (unstructured logs, no traces, metrics without labels) mean you're grepping text and guessing. Observability is not a place to save money — the cost of a 4-hour debug vs a 20-minute debug, multiplied by incident frequency, dwarfs the tooling bill. The choice is also about the team's ability to use the tools: the best tracing system is useless if no one knows how to read a flame graph.
When
At service creation, before the first incident. Takes 1-2 days to set up. Revisit if debugging is consistently slow.
How
1. Ensure the three pillars: - Metrics (Prometheus/Datadog): counters + histograms with labels for every dimension - Traces (Jaeger/DD Trace): distributed, sampled, with span attributes for business context - Logs (Loki/ELK): structured JSON, correlated to traces via trace_id 2. Require correlation: every log line has a trace_id; every trace has user_id + request_id 3. Pick managed (Datadog/New Relic) vs self-hosted (Prometheus/Loki) via build-vs-buy - Default managed for small teams; self-hosted at large scale (cost) 4. Instrument the business, not just the infra: "orders_created" not just "http_requests" 5. Set up dashboards per service: golden signals (latency, traffic, errors, saturation) 6. Test the observability: inject a fault, confirm you can find it via the tools in < 10 min
Example
Stack for a 5-service system:
Metrics: Datadog (managed, $1.5k/mo) — dashboards per service, SLO alerts
Traces: Datadog APM — sampled 10%, span attributes: user_id, order_id
Logs: Datadog Logs — structured JSON, auto-correlated to traces
Instrumentation:
logger.info("order_created", { order_id, user_id, total, trace_id })
tracer.span("charge_card", { user_id, amount })
Fault injection test: kill the DB → trace shows 5s span on DB →
log shows "connection_refused" → found in 3 minutes.Common Problems
- ⚠Logs without trace correlation — you have the error message but not the call path that led to it; correlation is what makes the three pillars actually useful.
Best Practices
- ✓Instrument at the framework boundary (middleware) so every request gets traces + metrics automatically — per-call instrumentation is forgotten on new endpoints.
- ✓Set SLOs (error rate, latency) with alerting, not just dashboards — a dashboard you don't look at is useless; an alert that pages you is acted on.