Deployment Gates & Progressive Delivery

Insert approval and verification gates between build and production so a deployment is a deliberate, validated step — not an automatic consequence of a merge.

Why

A pipeline that auto-deploys every merge to production is fast, but it turns every bug into a production incident — there is no human checkpoint between 'CI passes' and 'users are affected'. Deployment gates insert deliberate friction: an approval gate (a human reviews the change and clicks approve), a canary gate (the new version serves 1% of traffic and error rate is monitored for 10 minutes), and a rollback gate (if the canary's error rate exceeds 0.5%, the deployment auto-rolls back). This is progressive delivery: the deployment happens in stages (staging → canary → 10% → 50% → 100%), each stage is verified before the next, and any stage can fail without affecting all users. The trade-off is deployment speed (minutes become tens of minutes) for safety (a bad deployment affects 1% of users for 10 minutes, not 100% of users for an hour). For user-facing production systems, this trade-off is almost always worth it.

When

When the deployment target is a user-facing production environment — staging and preview environments can auto-deploy, but production should always have at least one gate (approval or canary).

How

1. Separate the build and deploy stages
   - CI pipeline: lint → test → build → publish artifact (to registry)
   - Deploy pipeline: trigger manually or on a tag, not on every merge

2. Add an approval gate before production
   - GitHub Environments: create a "production" environment with required reviewers
   - The deploy job pauses until a reviewer clicks "Approve" in the GitHub UI
   - This is a human checkpoint: the reviewer reads the diff, the changelog, and the test results

3. Deploy to staging first, then promote
   - Deploy to staging automatically (no approval needed)
   - Run smoke tests against staging
   - Promote to production only after staging smoke tests pass and a human approves

4. Add a canary stage for high-traffic services
   - Route 1% of traffic to the new version (Istio, Argo Rollouts, Cloudflare Workers)
   - Monitor error rate and latency for 10 minutes
   - If error rate < 0.5% and p99 latency < baseline, promote to 10% → 50% → 100%
   - If error rate >= 0.5%, auto-rollback and alert

5. Make rollback a one-click (or zero-click) operation
   - Keep the previous version's artifact for 24 hours
   - A rollback redeploys the previous artifact — no rebuild, no re-test
   - Auto-rollback on canary failure; manual rollback button for other cases

Example

# .github/workflows/deploy.yml — gated production deploy
name: deploy
on:
  push:
    tags: ["v*"]
jobs:
  deploy-staging:
    runs-on: ubuntu-latest
    steps:
      - run: ./deploy.sh staging
      - run: ./smoke-test.sh https://staging.example.com

  deploy-production:
    needs: deploy-staging
    environment:
      name: production        # requires reviewer approval
      url: https://example.com
    runs-on: ubuntu-latest
    steps:
      - run: ./deploy.sh production --canary 1%
      - run: ./canary-watch.sh --threshold 0.5% --duration 10m
      - run: ./deploy.sh production --promote 100%

Common Problems

⚠Auto-deploying every merge to production with no approval gate — a bad merge becomes a production incident within seconds, and the team develops a fear of merging that slows down development.
⚠Adding an approval gate but no canary — the human approves, the deployment goes to 100% of traffic at once, and a bug that the human missed affects all users immediately with no automatic rollback.
⚠Making rollback require a full rebuild and redeploy — rollback takes 15 minutes (the same as a forward deploy), so the team deploys forward fixes instead of rolling back, and incidents last longer than they should.
⚠Canary watch with too short a window (1 minute) — slow-burning bugs (a memory leak that crashes pods after 5 minutes) pass the canary and reach 100% before the error rate spikes.

Best Practices

✓Auto-deploy to staging on every merge, but require a human approval (GitHub Environment with required reviewers) for production — this gives the speed of CI for staging and the safety of a checkpoint for production.
✓Use a canary stage (1% → 10% → 50% → 100%) with an automated error-rate watch and auto-rollback — a bad deployment affects 1% of users for 10 minutes, not 100% for an hour.
✓Keep the previous artifact for 24 hours and make rollback a redeploy of that artifact (no rebuild) — rollback takes 30 seconds, so the team actually uses it instead of fixing forward.
✓Set the canary watch window to at least 10 minutes and monitor both error rate and p99 latency — this catches slow-burning bugs that a 1-minute window would miss.