Rollback Strategies for Zero-Downtime

Version retention, database reversibility, and automated rollback triggers for safe recovery.

CI/CDIntermediate

Why

When a deploy breaks production, every minute of downtime costs revenue and trust. A pre-tested rollback path — old version retained, DB migrations reversible, trigger automated — restores service in seconds instead of hours of 'fix-forward' panic.

Rollback must be a first-class deploy operation, not an afterthought.

When

Implement formal rollback for any production system with SLAs, frequent deploys (>1/day), or stateful data. Skip for static sites where redeploy is instant and stateless.

How

1. Retain versions — keep last N (3-5) image tags/deploys in registry + cluster, don't GC immediately 2. Reversible migrations — every up has a down; use expand/contract so old version still reads new schema 3. Feature flags — wrap risky changes in a flag, 'rollback' = toggle flag off (no redeploy, instant) 4. Automated trigger — monitor error rate/latency, auto-rollback on threshold breach (see canary-release) 5. Manual rollback — one command: kubectl rollout undo, vercel --prod --rollback, aws deploy rollback 6. Test rollback in staging — regularly practice rolling back, so the path is proven when you need it 7. Database rollback plan — for non-reversible migrations, snapshot before deploy, restore on rollback 8. Post-rollback — alert + postmortem, the rollback is not the end — find root cause before next deploy

Example

# K8s — retain revision history + instant rollback # deployment.yaml spec: revisionHistoryLimit: 5 # keep last 5 ReplicaSets for rollback # Rollback commands kubectl rollout undo deployment/myapp # to previous kubectl rollout undo deployment/myapp --to-revision=3 kubectl rollout status deployment/myapp # watch progress kubectl rollout history deployment/myapp # list revisions # Vercel — instant rollback to previous production deploy vercel --prod --rollback # CLI # or via API: POST /v13/deployments/{id}/rollback # Feature-flag rollback (no redeploy) — LaunchDarkly / Unleash import { LDClient } from "launchdarkly-node-server-sdk"; const client = await LDClient.init(process.env.LD_KEY); // "Rollback" = flip flag off, takes effect in <1s, no deploy await client.variation("new-checkout-flow", user, false); # Reversible DB migration (Knex) — always write down() exports.up = function (knex) { return knex.schema.table("orders", (t) => t.string("status_v2").nullable()); // expand }; exports.down = function (knex) { return knex.schema.table("orders", (t) => t.dropColumn("status_v2")); // contract }; # Auto-rollback trigger (GitHub Actions on deploy failure) - name: Deploy run: npm run deploy - name: Verify run: npm run smoke-test - name: Rollback on failure if: failure() run: npm run deploy:rollback

Common Problems

⚠Old version GC'd — registry retention too short, rollback target gone, forced to fix-forward
⚠Irreversible migration — added NOT NULL column, old version crashes on rollback, must fix-forward
⚠Rollback never tested — works in theory, fails in practice (config drift, missing env var)
⚠Fix-forward culture — team never rolls back, so the path rots and panic-fixing takes hours

Best Practices

✓Set revisionHistoryLimit: 5+ and registry retention 90 days — rollback target always available
✓Prefer feature flags over redeploy rollback — instant, no version juggling, no DB concerns
✓Run a 'rollback drill' monthly — deploy to staging, break it, roll back, verify the path works
✓Make every migration reversible — expand/contract, never destructive in the same deploy as code

FAQ

How fast should a rollback be?

Under 5 minutes — if it takes longer, users suffer while you investigate. Pre-bake the previous image/version so rollback is a single command (kubectl rollout undo). Never rebuild from an old commit during an incident.

What's the difference between rollback and fix-forward?

Rollback reverts to the last known-good version (fast, safe). Fix-forward deploys a new version that fixes the bug (slower, but keeps history clean). Rollback first to stop the bleeding, then fix-forward in a follow-up deploy.

How do I handle rollbacks with database changes?

Additive migrations (new columns) are rollback-safe. Destructive migrations (drop column) are not — you can't restore the data. Apply destructive changes in two phases: deploy code that stops writing, then drop the column after a safe period.

Why do I need to test my rollback?

Untested rollbacks fail when you need them most. Run a rollback in staging every release — it catches issues like incompatible DB schemas, missing images, or config that only exists in the new version. A rollback is a deploy; test it like one.

How do I automate rollback on error spikes?

Use Flagger or Argo Rollouts with a metric provider (Prometheus). Set: if error rate exceeds 1% for 2 min, auto-rollback. This catches issues in seconds vs. an on-call engineer taking 10 min to notice and act.