GitOps Rollbacks: When Reverting the Commit Is Not Enough

We reverted the merge at 02:40, watched Argo report Synced and Healthy within eleven seconds, and then watched the API keep returning 500. The configuration was back to what it had been. The database schema was not.

Git rollback restores declarations. It does not restore state, nor does it reach anything outside the repository.

Revert restores configuration, not state

A controller reconciles what is declared toward what exists. When you revert a commit, the declaration returns to its previous value and the controller acts on it. For a container image tag, a replica count, or an environment variable, that is a complete rollback.

For anything with a side effect, the declaration is only half the story.

bash

git revert <sha>
git push origin main
# Argo syncs, reports healthy, and the system is still broken

A green sync means the cluster matches the repository. It says nothing about whether the cluster works.

Three changes that do not undo themselves

Migrations. A forward migration added a column or renamed one. Reverting the code that writes to it does not drop the column or rename it back, and a system that now expects the old shape will fail against the new schema. This is the failure we hit, and it is the most common one.

Secrets and external services. Rotating a credential, creating a DNS record, or registering a webhook has effects in a system Git does not manage. The manifest goes back; the credential stays rotated and the old value stays invalid.

Cluster-scoped resources. CRDs, cluster roles, and webhooks are often applied by one application and used by others. Deleting or downgrading them can affect workloads that were never part of the change.

yaml

# the field that changes the question
syncPolicy:
  automated:
    prune: true
    selfHeal: true

With pruning on, reverting a commit does not merely change a value; it deletes whatever the previous commit had declared. That is correct behaviour and a poor moment to discover that a resource was shared.

Rollback as a rehearsed path

The fix is to decide, before the incident, which changes are reversible and which need a forward fix.

change	reverses cleanly	needs
image tag, replicas, env var	yes	revert
config map value	yes	revert plus restart
schema migration	no	a down migration, written at the same time
rotated credential	no	a stored fallback or a second rotation
CRD change	sometimes	a compatibility window

The column that matters is the one you write during the change, not during the failure. A migration ships with its inverse, or it ships behind a flag that keeps the old path alive until the new one is proven.

Progressive delivery buys time for exactly this. A canary that fails stops widening, and the rollback is a digest reference rather than a rebuild; the tradeoffs of each strategy are in Kubernetes deployment strategies.

The rehearsed sequence

For the reversible case I run the same four steps every time, so nobody is improvising:

bash

kubectl -n argocd get application api -o jsonpath='{.status.history[-1:]}'
argocd app history api
argocd app rollback api <revision>

argocd app rollback moves the application to a recorded revision, which is faster and more auditable than pushing a revert and waiting for the automated sync. Then the revert still gets pushed, so Git remains the source of truth and the next sync does not drift back.

Order matters here. Roll back the running system first to stop the bleeding, then make the repository agree. Doing it the other way around leaves the cluster broken for as long as the sync takes.

What I do at 3am

I stop the automated sync for the affected application, roll back to a known revision, confirm the user-visible symptom is gone, and only then look at whether state moved. Freezing first is the same instinct as break-glass handling: a controller that keeps applying a bad declaration will undo your manual fix within seconds.

Once the system is stable, the revert goes in as a normal pull request with a note about what does not reverse. That note is what the next person reads at 03:00.

Summary

A healthy sync means the cluster matches Git, nothing more. Reverts handle images, replicas, and configuration cleanly; they do not handle migrations, rotated credentials, or anything outside the repository. Decide reversibility when you ship the change, keep a forward fix ready for the rest, freeze automated sync before you intervene, and roll the running system back before you edit history.