GitOps on K8s: Argo CD in Production Without the Horror Stories

GitOps means the cluster state lives in git, and the only writer to production is the GitOps controller applying from git. No kubectl apply from laptops. No ad-hoc helm upgrade from CI. Here's the setup that survives production.

The repo structure

Two repos stay strictly separated:

App repos: application source code, with manifests or Helm charts committed.
Config/ops repo: the desired cluster state — every manifest, chart pin, and environment overlay.

Argo CD watches the config repo. When a PR merges, the controller detects drift and syncs. That single separation makes auditing trivial: "what changed in prod?" is always a git command.

Declare your apps as code

Apps-as-code beats clicking through the Argo CD UI:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: checkout-api
  namespace: argocd
spec:
  project: platform
  source:
    repoURL: https://github.com/acme/ledger.git
    targetRevision: main
    path: envs/prod/checkout-api
  destination:
    server: https://kubernetes.default.svc
    namespace: prod
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true

selfHeal and prune are what make drift die within seconds — and what makes a bad manifest apply automatically. Both warnings apply equally: review config PRs strictly and keep prod changes atomic.

Sync waves and hooks

Order matters in Kubernetes. Database migrations must run before the app rolls. Sync waves encode that. Argo CD installs the manifests in order, and waits for health before moving on. A failed migration rolls back the whole application unit — exactly what you want.

The three rules that prevent horror

Never let git diverge from cluster. selfHeal: true. If someone manually deletes a pod, the controller recreates it. That is the point — the only legitimate writers are Argo CD and the scheduler.
Pin your toolchains. Nothing in the ops repo is "latest." Every chart and image tag is pinned. latest in git is a lie.
Test before you let it self-heal. A selfHeal stream of a broken manifest is a circuit-breaker that never reads the manual. Vet chart rendering in CI (helm template + kubeconform dry-run) first.

Handling secrets

Never commit secrets. argocd-vault-plugin / External Secrets + a vault backend keep the secret reference in git and the value in the vault. Secrets rotation becomes a vault event, not a git event.

The gotchas we hit

Order of operations: create the namespace before workloads reference it (CreateNamespace=true or sync waves).
Sync looping: a controller that rewrites a field every reconcile shows up as a never-ending "OutOfSync" — add ignoreDifferences for those known churny fields.
Multi-cluster: keep one Argo CD per logical fleet or use projects and RBAC carefully — a misconfigured project is a cross-team security hole.
Humans still run, sometimes: allow break-glass access, logged and revoked, for the 2am case where git speed is too slow. Make it the exception, not the path of least resistance.

Summary

Git is the source of truth, Argo CD is the sole writer, self-heal keeps reality equal to git, and strict review is the safety net. Do that and "what changed in prod" becomes a boring question with a one-line answer.