Performance Monitoring

Continuously measure performance in production to catch regressions before users notice.

Why

Performance is not a one-time optimization; it is a property that degrades over time. Every new dependency, every new query, every new feature adds latency, and without monitoring the regression is invisible until users complain or bounce. Production monitoring with RUM (Real User Monitoring) shows the actual experience — not the synthetic Lighthouse score from a fast CI machine, but the 4-second LCP that real users on a mid-range phone over 3G actually see. APM (Application Performance Monitoring) traces the backend, showing which endpoint and which database query is slow. Together they give a complete picture: what the user experiences (RUM) and why it is slow (APM), and they alert on regressions so the team fixes them before they become a user-visible problem.

When

Set up RUM and APM from day one of production. Review dashboards weekly. Alert on regressions (p95 latency > SLO, LCP > 2.5s) in real time.

How

1. Set up RUM for frontend performance
   - web-vitals library → send LCP, INP, CLS, FCP, TTFB to analytics
   - Google Analytics 4, Vercel Analytics, or Datadog RUM
   - Break down by page, device type, geography, and connection speed
   - Look at p75, not average — the tail is where users suffer

2. Set up APM for backend performance
   - Datadog, New Relic, or OpenTelemetry + Jaeger/Grafana
   - Trace every request: entry point → service calls → DB queries → external APIs
   - Record the span duration for each step to find the bottleneck
   - Set up service maps and dependency maps to visualize the architecture

3. Define SLOs (Service Level Objectives) and alert on breaches
   - "p95 API latency < 200ms, 99.9% of requests, measured over 28 days"
   - "LCP < 2.5s for 75% of page loads"
   - Alert when the SLO is breached (not when a single request is slow)
   - Use error budgets: if you burn through the budget, stop shipping features and fix perf

4. Build dashboards that show the trends, not just the current value
   - Latency over time (is it getting worse?)
   - Error rate over time (are we trending toward an outage?)
   - Compare this week to last week, this deploy to the previous
   - Annotate deploys on the timeline to see which deploy caused a regression

5. Act on the monitoring
   - When an alert fires, trace the slow request to find the root cause
   - When a deploy regresses performance, roll back or fix forward immediately
   - Review the slowest endpoints quarterly and optimize the top 3

Example

// RUM: send web vitals to analytics
import { onLCP, onINP, onCLS } from "web-vitals";

const sendVital = ({ name, value, rating, id }) =>
  navigator.sendBeacon("/api/vitals", JSON.stringify({
    name, value, rating, id,
    page: location.pathname,
    device: navigator.userAgent,
    connection: navigator.connection?.effectiveType,
  }));

onLCP(sendVital); onINP(sendVital); onCLS(sendVital);

// APM: OpenTelemetry tracing
import { trace } from "@opentelemetry/api";

const tracer = trace.getTracer("my-app");
async function handleOrder(req, res) {
  return tracer.startActiveSpan("handleOrder", async (span) => {
    try {
      const order = await tracer.startActiveSpan("db.insert", () =>
        db.query("INSERT INTO orders ...", [req.body]),
      );
      span.setAttribute("order.id", order.id);
      res.json(order);
    } catch (err) {
      span.recordException(err);
      span.setStatus({ code: 2, message: err.message });
      throw err;
    } finally {
      span.end();
    }
  });
}

// SLO alert (Datadog monitor JSON)
{
  "name": "API p95 latency SLO breach",
  "query": "p95:trace.http.request.duration{service:api} > 200",
  "thresholds": { "critical": 200, "warning": 180 },
  "timeframe": "5m",
  "message": "API p95 latency exceeded 200ms SLO. Investigate slow endpoints."
}

Common Problems

⚠Monitoring only in CI with synthetic tests (Lighthouse) and not in production with RUM — the CI score is green on a fast machine, but real users on slow phones see 5-second loads that the monitoring never shows.
⚠Alerting on average latency instead of p95/p99 — the average is 100ms, the alert never fires, but 5% of users see 3-second requests and are bouncing.
⚠Dashboards that show the current value but not the trend — the latency is 150ms today, which looks fine, but it was 80ms last month and is trending toward the SLO breach.
⚠Monitoring that alerts but nobody investigates — the alert fires at 3am, the on-call engineer silences it, and the regression stays for weeks because there is no follow-up process.

Best Practices

✓Use RUM (web-vitals → analytics) for the real user experience and APM (OpenTelemetry → Jaeger) for the backend trace — together they show what the user sees and why it is slow.
✓Define SLOs based on user impact (p95 latency, 99.9% availability) and alert on SLO breach, not on individual slow requests — the SLO is the contract with the user, and breaching it is the signal to act.
✓Show trends on dashboards (this week vs last week, with deploy annotations) so regressions are visible as a step change on the timeline, not just a number that is slightly higher today.
✓Treat every performance alert as a ticket with an owner and a deadline — an alert that is silenced and forgotten is worse than no monitoring, because it creates the illusion of coverage.