Releases & Rollback

How we ship to production and how to roll back. Applies to the Kamal-deployed services (superset-bi, ai-ml-platform). Build once on develop, validate on dev, then promote the exact image to prod — prod runs the bytes you tested, never a fresh build.

flowchart LR
  m["merge to develop"] --> b["Deploy workflow:<br/>build :sha + deploy DEV"]
  b --> v["validate on dev"]
  v --> t["git tag vX.Y.Z"]
  t --> p["Release: promote<br/>:sha → :vX.Y.Z"]
  p --> d["deploy-prod<br/>kamal deploy --skip-push"]

Image tags

Tag	Meaning	Mutable?
`:<sha>`	exact build; the rollback target	❌
`:vX.Y.Z`	a release (pin prod to this)	❌
`:vX.Y`	rolling "latest patch of this minor"	✅
`:latest`	newest dev build — never pin prod to it	✅

Cut a release

git tag v1.2.3 <validated-commit>     # a commit already built+deployed on dev
git push origin v1.2.3

The Release workflow then: 1. promote — retags :<sha> → :v1.2.3 + :v1.2 (no rebuild). 2. deploy-prod — waits for the production environment approval (Required Reviewers), then kamal deploy --skip-push to prod.

Don't tag an unbuilt commit

promote retags the image built for that commit's SHA on develop. If that SHA was never built, it fails with a clear error — tag a commit that went through the dev deploy.

Rollback

Is rollback automatic? Mostly no — but there's a safety net

Kamal (v2) does not auto-revert after a bad deploy. But its deploy runs a health check: if the new container fails it, Kamal aborts and never switches the proxy — the old healthy version keeps serving, so a broken image causes no downtime. That covers "the build is broken." It does not cover "the deploy passed health checks but the release is bad" — that's a manual rollback.

Manual rollback (the command)

Kamal keeps the previous container on the host, so rollback is fast. Run it on the runner/host where Kamal + the deploy config live (or via kamal over SSH):

kamal rollback <previous-version> -c config/deploy.yml
# ai-ml is per-service: kamal rollback <ver> -c config/deploy.colab.yml   (repeat per service)

<previous-version> = the image tag of the last good release — the prior :<sha>. The :vX.Y.Z tags tell you which SHA each release was (v1.2.2 ↔ its commit SHA).
Kamal reboots that version's container, pulling it from DOCR if it was pruned locally. Because we keep :<sha> + :vX.Y.Z tags (and garbage collection never removes tagged images), the rollback target is always retrievable.

Rollback via the pipeline (audited path)

Prefer an approval/audit trail? Re-point prod at the previous :vX.Y.Z and redeploy it (re-run deploy-prod for that tag). Same result, through the same production-environment gate.

After a rollback

Record the SHA you rolled from and to (see Incident Response).
Fix forward: cut a new vX.Y.(Z+1) rather than leaving prod pinned to an old release.

Why no auto-rollback?

Auto-reverting after a deploy that passed health checks needs external monitoring (the app is healthy at deploy time, bad later) plus a trigger — Kamal doesn't watch post-deploy. The health-check gate + manual kamal rollback is the standard Kamal posture. If you want auto-revert later, wire prod alerts (Sentry / uptime checks) to a rollback action.

Prerequisites for prod deploys

deploy-prod is dormant until the production GitHub environment is configured (its own DB host/password, a fresh SUPERSET_SECRET_KEY, OAuth creds, DEPLOY_HOST, etc.) and Required Reviewers are set on it. See the per-repo release.yml job comments for the exact list.