Skip to content
Back to Maintenance & LTS

Rollback Plan

Make every upgrade instantly reversible — the safety net that lets you upgrade aggressively without fear.

Why

An upgrade you can't roll back is a bet-the-company event, so teams avoid upgrading — which compounds debt. A rollback plan converts an upgrade from a one-way door to a two-way door: if staging or canary shows regressions, you revert in seconds and try again next week. The key insight is that rollback must be rehearsed before you need it; an unrehearsed rollback fails at 3am because the old image was garbage-collected, the DB migration isn't reversible, or the feature flag was removed. Rehearsing in staging every release makes rollback a reflex, not a research project.

When

Before every major upgrade deploy. Rehearse the rollback in staging as part of the upgrade PR's definition of done.

How

1. Keep the previous image: tag the current prod image as :prev before deploying the upgrade
2. For DB changes: only ship forward-compatible migrations (expand-contract) so the old image still works on the new schema
3. Deploy the upgrade behind a feature flag or as a canary (10% traffic)
4. Watch the golden signals (error rate, p95 latency, key business metric) for 10 min
5. If any signal degrades beyond threshold, rollback: kubectl rollout undo (or toggle the flag off)
6. Rehearse: in staging, deploy the upgrade then immediately roll back — confirm the app still works
7. Document the rollback command in the upgrade PR description so on-call doesn't have to invent it

Example

# Tag previous image before upgrade deploy
docker tag registry/app:current registry/app:prev
docker push registry/app:prev

# Deploy the upgrade as a canary
kubectl set image deployment/app app=registry/app:upgraded
kubectl rollout status deployment/app  # wait for canary

# If error rate spikes, rollback in <30s
kubectl rollout undo deployment/app

# Feature-flag rollback (instant, no redeploy)
# In your flag system:
curl -X POST https://flags/api/v1/flags/use-react-18 -d '{"enabled":false}'

# Rehearse in staging (CI job on the upgrade PR):
#   1. deploy :upgraded to staging
#   2. run smoke tests
#   3. kubectl rollout undo
#   4. run smoke tests again — must still pass

Common Problems

  • ⚠The old image was garbage-collected by the time you need it — rollback fails because the artifact is gone.
  • ⚠A DB migration ran with the upgrade and isn't reversible — you can roll back the code but the schema is stuck.
  • ⚠Rollback was never rehearsed, so at 3am on-call discovers the command is wrong or the flag was hard-coded out.

Best Practices

  • ✓Always tag the previous image as :prev and retain it for at least 24h after an upgrade deploy.
  • ✓Use expand-contract DB migrations so the old code version keeps working on the new schema — rollback then needs no DB revert.
  • ✓Make rollback rehearsal a CI gate on every upgrade PR: deploy, undo, re-test — if rollback fails, the upgrade doesn't ship.