Rollbacks: when the deploy goes wrong

Deploys fail, so plan for it

Some bug will pass the tests and reach production. The professional metric is not never breaking, it is time to recover, and containers make recovery unusually clean.

A rollback is simply deploying the previous image tag. Because lesson 7-3 tagged every image with its commit hash and the registry from lesson 6-1 keeps them all, last week's known-good build is still sitting there, byte-identical.

docker pull ghcr.io/ada/shop:41a9c02   # the previous good build
docker stop web && docker rm web
docker run -d --name web -p 80:8000 ghcr.io/ada/shop:41a9c02

No rebuilding under pressure and no reverting commits at 3am first. Restore service in seconds, then fix the code calmly.

This only works if tags are immutable, meaning never overwritten, which is the real argument against deploying :latest from lesson 6-1.

An incident drill in bash

v4 just shipped and is broken, so the rollback target is the release before it.

The starting point:

releases="v1 v2 v3 v4"
current="v4"
rollback=""

Filling in the target and printing the two status lines:

releases="v1 v2 v3 v4"
current="v4"
rollback="v3"
echo "deployed:    myapp:$current"
echo "rolling back to: myapp:$rollback"

Output

deployed:    myapp:v4
rolling back to: myapp:v3

The previous release before v4 in the list is v3, which is the only piece of judgment in the drill. Real rollback scripts read that value from a deployment record rather than a hardcoded list, precisely so nobody has to guess it during an incident.

Deploying without downtime

Stopping the old container before starting the new one leaves a gap where users see errors. Two standard patterns avoid it.

Rolling deploy. Run several identical app containers behind a load balancer, a small server that receives all incoming traffic and spreads it across the healthy containers, which is often exactly what nginx from unit 2 is doing. Replacing the containers one at a time means users are always served by the remaining ones, and old and new versions briefly serve together.

Blue-green. Run a full new copy, green, next to the current one, blue, test it, then flip traffic all at once. Rollback is flipping back, which is instant.

Both patterns depend on the platform knowing whether the new version is actually healthy before sending it traffic. That signal is the health check, and it is the centerpiece of lesson 9-2.

The first move during an outage

The fastest professional first move is to redeploy the previous image tag from the registry, then debug calmly.

The previous tag is a tested, immutable artifact already sitting in the registry, so redeploying it takes seconds and restores users first.

Rebuilding under pressure introduces new untested bytes, and debugging live while users see errors burns the only metric that matters during an incident. Diagnosis is much easier once the site is back up and the pressure is gone.

Rolling back a blue-green flip

When traffic is flipped to green and users hit a bug, the rollback action is to flip traffic back to blue.

The old environment was never torn down, so blue is still running untouched, and the rollback is the same operation as the deploy in reverse. That makes it effectively instant and close to risk-free.

That safety is why teams accept the cost of blue-green, which is briefly running two full copies of the application. The one thing it does not undo is a database migration, which is why schema changes are made backward-compatible before the flip.