Zero-Downtime Deployments in Production: Kubernetes Rolling Updates & Migrations

A concrete engineering blueprint to eliminate 502 Bad Gateway glitches, manage graceful pod shutdowns, and execute safe zero-downtime database schema migrations.

Executive Summary

PreStop hooks, SIGTERM handling, expand/contract database migration patterns, and ArgoCD canary analysis for zero-downtime releases.

Author: Ayesha Tariq, CTO & Cloud Lead

The Root Cause of 502 Glitches During Kubernetes Deployments

Most teams assume that setting maxUnavailable: 0 in their Kubernetes Deployment manifest guarantees zero downtime. Yet users still experience intermittent 502 Bad Gateway errors during rollouts. Why? Because when Kubernetes decides to terminate an old pod, it sends the SIGTERM signal to the application container simultaneously with removing the pod IP from the Ingress/Service endpoint list. Ingress controllers continue routing active traffic for 200-800ms to a pod that has already started shutting down its HTTP server.

The Fix: PreStop Hooks & Graceful Signal Handling

By adding a preStop hook containing a sleep 5 command to your container spec, you force the pod to wait 5 seconds before terminating its process. This gives the Ingress controller and kube-proxy plenty of time to drain and deregister the IP from active load balancing pools. Combined with Node.js/Go graceful server.close() listeners that finish in-flight requests, 502 errors drop to absolute zero.

Zero-Downtime Database Migrations (The Expand/Contract Pattern)

Never rename or drop a database column in a single migration step! If an old pod is still running while the new migration executes, queries fail instantly. Instead, use the 3-phase Expand/Contract pattern: 1) Expand: Add the new column as nullable and write to both. 2) Backfill: Copy legacy data in background batches. 3) Contract: Switch reads to the new column, verify in production, and drop the legacy column in a subsequent sprint.

Deployment Hardening Rules

  • Always define both readinessProbe and livenessProbe with realistic initialDelaySeconds and failureThresholds.
  • Set resource limits and requests accurately to prevent CPU throttling and OOMKilled evictions during traffic surges.
  • Use PodDisruptionBudgets (PDB) to prevent cluster autoscalers from draining multiple pods simultaneously during node upgrades.
  • Implement automated Canary analysis using Argo Rollouts with Prometheus metric thresholds.