Load Testing

Apply realistic and peak traffic to find the throughput ceiling and the first thing that breaks.

Why

Performance issues do not show up at low traffic. A query that takes 50ms with one user takes 50 seconds when 1000 concurrent users hit the same unindexed column, and the database connection pool exhausts at 200 concurrent requests, not at 10. Load testing is the only way to find these ceilings before your users do — at launch, at a marketing push, at a viral moment. The goal is not to produce a big number for a slide; it is to find the first bottleneck, understand why it breaks, and decide whether to fix it or to provision around it with autoscaling and rate limiting.

When

Before any launch, before any marketing event that could spike traffic, and as a periodic baseline (monthly) to catch performance drift. Also after any major architecture change (new cache layer, database migration).

How

1. Define the load profile from real analytics, not guesses
   - Pull the p50, p95, p99 request rates from your APM for the last 30 days
   - Identify the peak hour and the endpoint mix (60% GET /products, 20% POST /orders...)
   - Set the target: "sustain 2x peak for 10 min with p95 < 500ms"

2. Choose a tool that can generate the volume
   - k6 (JS, developer-friendly, good for 10k+ VUs) or Locust (Python)
   - Gatling (Scala, rich reports) for enterprise scenarios
   - Run from a machine that can actually generate the load (not your laptop)

3. Ramp up gradually, do not slam the system
   - 0 → 100 VUs over 2 min, hold 100 for 5 min, ramp to 500 over 2 min, hold
   - A sudden spike tells you nothing — you cannot tell the ceiling from the cold-start

4. Monitor the system under load, not just the test results
   - Watch CPU, memory, DB connections, cache hit rate, error rate, p99 latency
   - The first metric to spike is your bottleneck; note the VU count at which it happens

5. Find the ceiling, then stop and analyze
   - Increase load until error rate > 1% or p95 > target — that is the ceiling
   - Profile the bottleneck (slow query, connection pool, GC pause, lock contention)
   - Fix or provision, then re-test to confirm the ceiling moved

Example

// k6 script: load-test-checkout.js
import http from "k6/http";
import { check, sleep } from "k6";
import { Rate, Trend } from "k6/metrics";

const errorRate = new Rate("checkout_errors");
const latency = new Trend("checkout_latency");

export const options = {
  stages: [
    { duration: "2m", target: 100 },   // ramp to 100 VUs
    { duration: "5m", target: 100 },   // hold 100
    { duration: "2m", target: 500 },   // ramp to 500
    { duration: "5m", target: 500 },   // hold 500 — find the ceiling
    { duration: "2m", target: 0 },     // ramp down
  ],
  thresholds: {
    http_req_duration: ["p(95)<500"],  // 95% of requests under 500ms
    checkout_errors: ["rate<0.01"],    // < 1% errors
  },
};

export default function () {
  const res = http.post("https://staging.example.com/api/checkout", JSON.stringify({
    product_id: 42, qty: 1, token: __ENV.TEST_TOKEN,
  }), { headers: { "Content-Type": "application/json" } });

  errorRate.add(res.status !== 200);
  latency.add(res.timings.duration);
  check(res, { "status 200": (r) => r.status === 200 });
  sleep(1); // 1s think time between iterations
}

// Run: k6 run --env TEST_TOKEN=xxx load-test-checkout.js

Common Problems

⚠Running the load test from a single laptop that cannot generate the target volume — the test reports "5000 RPS achieved" but the laptop's network was the bottleneck, and the real system could handle 50k.
⚠Using uniform random data instead of realistic distributions means every request hits a different product, so the cache is never warm and you measure cold-read performance instead of the real mixed workload.
⚠Slamming the system with 1000 VUs instantly instead of ramping up — you see a crash but cannot tell whether it was the cold start (connection pool init), the steady-state load, or a genuine ceiling.
⚠Testing only the happy path and never the error paths (payment declined, out of stock) means the error-handling code paths are untested under load and may themselves be the bottleneck.

Best Practices

✓Run the load generator from a machine with enough CPU and network bandwidth — a cloud VM in the same region as the target, or a distributed k6 cluster for 10k+ VU tests.
✓Derive the load profile from real APM data (request mix, peak hour, think time) so the test reproduces the actual production workload, not a synthetic uniform hit.
✓Monitor the target system's internals (DB connections, cache hit rate, GC time) during the test — the test result tells you it broke, the monitoring tells you why.
✓Treat the first load test as a baseline measurement, not a pass/fail — record the ceiling and the bottleneck, then re-test after each optimization to confirm improvement.