managing-deployment-rollbacks

Overview

Implement and execute deployment rollback procedures for Kubernetes, ECS, Lambda, and cloud VM deployments. Detect failed deployments via health checks and error rate monitoring, then automatically or manually revert to the last known good version with minimal downtime and data integrity preservation.

Prerequisites

• kubectl configured with cluster access and permission to manage deployments
• Deployment history retained (Kubernetes revisionHistoryLimit, ECS task definition versions)
• Monitoring system tracking error rate, latency, and health check status (Prometheus, Datadog, CloudWatch)
• Previous deployment artifacts (container images, task definitions) still available in the registry
• Database migration strategy that supports backward compatibility (expand-contract pattern)

Instructions

1. Detect deployment failure: monitor error rate, P99 latency, pod restart count, and health check responses for 5-10 minutes post-deploy
2. Assess rollback scope: determine if the issue is application code, configuration, or infrastructure
3. For Kubernetes: execute kubectl rollout undo deployment/<name> to revert to the previous revision
4. For ECS: update the service to use the previous task definition revision
5. For Lambda: point the alias back to the previous function version
6. Verify database compatibility: ensure the previous application version works with the current database schema (no forward-only migrations were applied)
7. Confirm rollback success: verify health checks pass, error rate returns to baseline, and user-facing functionality is restored
8. Generate a post-incident report: document what failed, when rollback was triggered, time to recovery, and root cause
9. Create automated rollback rules: configure Kubernetes readiness probes, Argo Rollouts analysis, or CloudWatch alarms to trigger rollback without manual intervention

Output

• Rollback scripts for each deployment target (Kubernetes, ECS, Lambda)
• Automated rollback configuration (Kubernetes probes, Argo Rollouts AnalysisTemplate)
• Post-incident report template with timeline and root cause sections
• Monitoring dashboard with rollback trigger indicators
• Database migration rollback procedures (down migrations, backward-compatible schemas)

Error Handling

| Error | Cause | Solution | |-------|-------|---------| | no rollout history found | Revision history limit set to 0 or deployment was created fresh | Increase revisionHistoryLimit in deployment spec; manually specify the target image tag | | Rollback succeeded but errors persist | Issue is in configuration or external dependency, not application code | Check ConfigMaps, Secrets, and external service health; rollback configuration changes separately | | Database schema incompatible after rollback | Forward-only migration applied during failed deployment | Apply a down migration or use expand-contract pattern; never deploy breaking schema changes alongside code | | Old image no longer in registry | Lifecycle policy deleted the previous image | Restore from backup or rebuild from the git tag; extend image retention for production tags | | Rollback causes service disruption | Insufficient replicas during rollback transition | Set maxUnavailable: 0 in rolling update strategy to ensure zero-downtime rollback |

Examples

• "Roll back the production Kubernetes deployment to the previous revision after detecting a spike in 5xx errors."
• "Create an automated rollback policy using Argo Rollouts that reverts if error rate exceeds 1% during the first 10 minutes after deploy."
• "Generate a rollback runbook for an ECS service that includes steps to revert task definition, validate health, and notify the team via Slack."

Resources

• Kubernetes rollout management: https://kubernetes.io/docs/concepts/workloads/controllers/deployment/#rolling-back-a-deployment
• Argo Rollouts analysis: https://argoproj.github.io/argo-rollouts/features/analysis/
• AWS ECS rolling updates: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/deployment-type-ecs.html
• Database migration patterns: https://martinfowler.com/articles/evodb.html