Part 3 of the data-e2e series. Issue numbers refer to the deployment log; the full procedure is the Argo CD plan §0.

Most Argo CD tutorials start with an empty cluster. Mine had 28 Helm releases and raw manifests, running a lakehouse that took days to get stable on slow hardware. The goal was to make Git the only way to change the cluster, without redeploying anything. If Argo CD’s first sync restarted a database on that disk, I’d be back to Part 1.

It worked: 205 resources adopted, no pod replaced, and automated sync with prune and self-heal turned on at the end. Here is how, and the four things that would have gone wrong.

The plan in one paragraph

1. Install Argo CD and Sealed Secrets.
2. Generate one Application per component, pointing at the same charts, versions and values files the deploy scripts used, and commit them on a branch Argo CD tracks.
3. Create every Application with sync disabled. Then for each app: – look at the diff; – make Git match the cluster until the diff is empty, or only Argo CD’s own tracking annotation; – sync; – check that no pod restarted; – move to the next app.
4. Only when every app is clean, turn on automation.

The important part is that the adoption is driven by the diff, not by hope.

Problem 1: secrets that only existed on the command line

The deploy scripts passed secrets to Helm with --set, so the values files in Git contained placeholders such as password: "set-at-install". That works for scripts. It is a disaster for Argo CD, which renders only what is in Git. Its first sync would have replaced six real passwords with placeholders.

I know because I had already done it once by hand (#90). A helm upgrade --reuse-values -f values.yaml let the file’s placeholder override the reused secret, and JupyterHub’s password silently became set-at-install. Nothing crashed; logins just started failing.

Fix (#103): every secret moved out of Helm values into a Kubernetes Secret that the chart references (existingSecret-style). Each Secret was then sealed with Sealed Secrets and committed.

• A SealedSecret is encrypted with the cluster’s public key. It is safe in Git, and only the in-cluster controller can decrypt it.
• The seal script copies only type and data, pipes straight into kubeseal, and never writes plaintext to disk.
• 30 Secrets were sealed with byte-identical data, verified by hash before adoption.
• The deploy scripts changed from “create or update secret” to “create only if missing”, so a script run can no longer rotate a password behind Git’s back.

One thing you must not skip: back up the sealing key. If the cluster is rebuilt without it, every SealedSecret in Git becomes undecryptable.

Problem 2: values that are random on purpose

Some charts generate secrets at install time with Helm’s lookup function: “if the Secret already exists, reuse its value, otherwise generate one”. Examples are JupyterHub’s proxy token and cookie secret, and Bitnami’s postgres-password.

Argo CD renders charts with helm template, which has no cluster access, so lookup always returns nothing and the chart generates a new random value on every render (#104). The result is a permanent diff, and with self-heal, a password change on every sync.

Fix, in order of preference:

1. If the chart supports an existing Secret, create one and seal it.
2. Otherwise, tell Argo CD to ignore exactly those fields with ignoreDifferences (the generated keys and any checksum annotation derived from them), and set RespectIgnoreDifferences=true so a sync never writes them either.

I verified the second option the hard way: JupyterHub’s keys were unchanged after a sync and the hub did not restart. The cost is that ignored fields are invisible to diffs, which is acceptable for random tokens.

Problem 3: Helm features Argo CD doesn’t have

• Post-renderers (#105). Opik needed a Helm post-renderer to add startup probes that its MySQL sub-chart hard-codes away (Part 1). Argo CD has no post-renderer hook, so Opik became a Kustomize app: helmCharts: inflates the chart, and JSON patches add the probes. The same patches stay in the script path, so the two renderings must be kept in step.
• Helm hooks (#107). Argo CD understands helm.sh/hook only for Helm sources. A chart inflated by Kustomize turned its MinIO bucket job into an ordinary resource that went “missing” after its TTL deleted it. That left the app permanently OutOfSync. Argo CD’s own hook annotations (argocd.argoproj.io/hook: PostSync) fixed it for Kustomize, and for charts with TTL’d migration Jobs (Airflow) too.
• Huge CRDs (#102). The Spark Operator’s CRD is bigger than the 256 KB annotation limit that client-side apply uses to store last-applied-configuration. ServerSideApply=true for that app (and for Strimzi, KServe, Opik) avoids the annotation entirely.

Problem 4: the network, again

The repo-server could not fetch the repository: git fetch … timeout after 1m40s (#100). The pods are IPv4-only and GitLab answers IPv6 first. An SSH config with AddressFamily inet fixed it (more in Part 2).

The adoption itself

After all that, the diffs for 205 resources were boring. The only differences were Argo CD’s tracking annotation and one Helm label. Each app was synced individually, and after each one kubectl get pods was checked for restarts.

Then automation was turned on: automated: {prune: true, selfHeal: true} on all 28 apps.

The self-heal test is the satisfying part. I changed a Deployment by hand, and Argo CD reverted it in 10 seconds. From that moment, kubectl edit is a temporary experiment, and Git is the write path.

What changed in day-to-day work

• A change is a commit. Values, manifests and Sealed Secrets go to main, and Argo CD applies them within 5 minutes. There’s no webhook from GitLab into the LAN, so it polls.
• The scripts still exist, for bootstrap and break-glass. They never overwrite existing Secrets, so they can’t fight Argo CD.
• Drift is visible. “OutOfSync” now means someone changed the cluster outside Git. With self-heal, it is also already undone.

If you’re about to do this

1. Inventory every --set. Each one is a future Argo CD regression.
2. Find every lookup in your charts. Grep the rendered templates for random-looking values that change between two helm template runs.
3. Adopt with sync off and drive it by diff, one app at a time.
4. Seal secrets from the live cluster (byte-identical), not from a .env file that may have drifted. Back up the sealing key somewhere that isn’t the cluster.
5. Expect to translate post-renderers and hooks for Argo CD.
6. Turn on self-heal last, after an empty diff, and test it.

Previous: Part 2 (Networking · Next: Part 4) Upgrading with no spare capacity.

---

Get the next one in your inbox

Enjoyed this? I write about local AI, self-hosting, and cloud cost ninjutsu. Join the Ninja Dispatch for occasional, high-signal notes. No fluff.

No spam. Unsubscribe anytime.