{"id":3521,"date":"2026-06-02T16:12:00","date_gmt":"2026-06-02T22:12:00","guid":{"rendered":"https:\/\/yyc.ninja\/index.php\/2026\/10\/20\/adopting-a-running-kubernetes-cluster-into-gitops-without-restarts\/"},"modified":"2026-10-23T11:26:17","modified_gmt":"2026-10-23T17:26:17","slug":"adopting-a-running-kubernetes-cluster-into-gitops-without-restarts","status":"publish","type":"post","link":"https:\/\/yyc.ninja\/index.php\/2026\/06\/02\/adopting-a-running-kubernetes-cluster-into-gitops-without-restarts\/","title":{"rendered":"Adopting a running cluster into GitOps without restarting a single pod"},"content":{"rendered":"
\"Adopting<\/figure>\n

Part 3 of the data-e2e series. Issue numbers refer to the deployment log; the full procedure\nis the Argo CD plan \u00a70.<\/em><\/p>\n

Most Argo CD tutorials start with an empty cluster. Mine had 28 Helm releases and raw manifests, running a lakehouse\nthat took days to get stable on slow hardware. The goal was to make Git the only way to change the cluster<\/strong>,\nwithout redeploying anything. If Argo CD’s first sync restarted a database on that disk, I’d be back to Part 1.<\/p>\n

It worked: 205 resources adopted, no pod replaced, and automated sync with prune and self-heal turned on at the end.\nHere is how, and the four things that would have gone wrong.<\/p>\n

The plan in one paragraph<\/h3>\n
    \n
  1. Install Argo CD and Sealed Secrets.<\/li>\n
  2. Generate one Application<\/code> per component, pointing at the same<\/em> charts, versions and values files the deploy\n scripts used, and commit them on a branch Argo CD tracks.<\/li>\n
  3. Create every Application with sync disabled<\/strong>. Then for each app:\n – look at the diff;\n – make Git match the cluster until the diff is empty, or only Argo CD’s own tracking annotation;\n – sync;\n – check that no pod restarted;\n – move to the next app.<\/li>\n
  4. Only when every app is clean, turn on automation.<\/li>\n<\/ol>\n

    The important part is that the adoption is driven by the diff<\/em>, not by hope.<\/p>\n

    Problem 1: secrets that only existed on the command line<\/h3>\n

    The deploy scripts passed secrets to Helm with --set<\/code>, so the values files in Git contained placeholders such as\npassword: \"set-at-install\"<\/code>. That works for scripts. It is a disaster for Argo CD, which renders only what is in\nGit<\/strong>. Its first sync would have replaced six real passwords with placeholders.<\/p>\n

    I know because I had already done it once by hand (#90). A helm upgrade --reuse-values -f values.yaml<\/code> let the\nfile’s placeholder override the reused secret, and JupyterHub’s password silently became set-at-install<\/code>. Nothing\ncrashed; logins just started failing.<\/p>\n

    Fix (#103):<\/strong> every secret moved out of Helm values into a Kubernetes Secret that the chart references<\/em>\n(existingSecret<\/code>-style). Each Secret was then sealed with Sealed Secrets and committed.<\/p>\n

      \n
    • A SealedSecret is encrypted with the cluster’s public key. It is safe in Git, and only the in-cluster controller\n can decrypt it.<\/li>\n
    • The seal script copies only type<\/code> and data<\/code>, pipes straight into kubeseal<\/code>, and never writes plaintext to disk.<\/li>\n
    • 30 Secrets were sealed with byte-identical data, verified by hash before adoption.<\/li>\n
    • The deploy scripts changed from “create or update secret” to “create only if missing”<\/strong>, so a script run can no\n longer rotate a password behind Git’s back.<\/li>\n<\/ul>\n

      One thing you must not skip: back up the sealing key<\/strong>. If the cluster is rebuilt without it, every SealedSecret in\nGit becomes undecryptable.<\/p>\n

      Problem 2: values that are random on purpose<\/h3>\n

      Some charts generate secrets at install time with Helm’s lookup<\/code> function: “if the Secret already exists, reuse its\nvalue, otherwise generate one”. Examples are JupyterHub’s proxy token and cookie secret, and Bitnami’s\npostgres-password<\/code>.<\/p>\n

      Argo CD renders charts with helm template<\/code>, which has no cluster access, so lookup<\/code> always returns nothing and the\nchart generates a new random value on every render<\/strong> (#104). The result is a permanent diff, and with self-heal,\na password change on every sync.<\/p>\n

      Fix, in order of preference:<\/strong><\/p>\n

        \n
      1. If the chart supports an existing Secret, create one and seal it.<\/li>\n
      2. Otherwise, tell Argo CD to ignore exactly those fields with ignoreDifferences<\/code> (the generated keys and<\/em> any\n checksum annotation derived from them), and set RespectIgnoreDifferences=true<\/code> so a sync never writes them\n either.<\/li>\n<\/ol>\n

        I verified the second option the hard way: JupyterHub’s keys were unchanged after a sync and the hub did not restart.\nThe cost is that ignored fields are invisible to diffs, which is acceptable for random tokens.<\/p>\n

        Problem 3: Helm features Argo CD doesn’t have<\/h3>\n
          \n
        • Post-renderers (#105).<\/strong> Opik needed a Helm post-renderer to add startup probes that its MySQL sub-chart\n hard-codes away (Part 1). Argo CD has no post-renderer hook, so Opik became a Kustomize<\/strong> app: helmCharts:<\/code>\n inflates the chart, and JSON patches add the probes. The same patches stay in the script path, so the two\n renderings must be kept in step.<\/li>\n
        • Helm hooks (#107).<\/strong> Argo CD understands helm.sh\/hook<\/code> only for Helm sources. A chart inflated by Kustomize\n turned its MinIO bucket job into an ordinary resource that went “missing” after its TTL deleted it. That left the\n app permanently OutOfSync. Argo CD’s own hook annotations (argocd.argoproj.io\/hook: PostSync<\/code>) fixed it for\n Kustomize, and for charts with TTL’d migration Jobs (Airflow) too.<\/li>\n
        • Huge CRDs (#102).<\/strong> The Spark Operator’s CRD is bigger than the 256 KB annotation limit that client-side apply\n uses to store last-applied-configuration<\/code>. ServerSideApply=true<\/code> for that app (and for Strimzi, KServe, Opik)\n avoids the annotation entirely.<\/li>\n<\/ul>\n

          Problem 4: the network, again<\/h3>\n

          The repo-server could not fetch the repository: git fetch \u2026 timeout after 1m40s<\/code> (#100). The pods are IPv4-only and\nGitLab answers IPv6 first. An SSH config with AddressFamily inet<\/code> fixed it (more in Part 2).<\/p>\n

          The adoption itself<\/h3>\n

          After all that, the diffs for 205 resources were boring. The only differences were Argo CD’s tracking annotation and\none Helm label. Each app was synced individually, and after each one kubectl get pods<\/code> was checked for restarts.<\/p>\n

          Then automation was turned on: automated: {prune: true, selfHeal: true}<\/code> on all 28 apps.<\/p>\n

          The self-heal test is the satisfying part. I changed a Deployment by hand, and Argo CD reverted it in 10 seconds<\/strong>.\nFrom that moment, kubectl edit<\/code> is a temporary experiment, and Git is the write path.<\/p>\n

          What changed in day-to-day work<\/h3>\n
            \n
          • A change is a commit.<\/strong> Values, manifests and Sealed Secrets go to main<\/code>, and Argo CD applies them within 5\n minutes. There’s no webhook from GitLab into the LAN, so it polls.<\/li>\n
          • The scripts still exist<\/strong>, for bootstrap and break-glass. They never overwrite existing Secrets, so they can’t\n fight Argo CD.<\/li>\n
          • Drift is visible.<\/strong> “OutOfSync” now means someone changed the cluster outside Git. With self-heal, it is also\n already undone.<\/li>\n<\/ul>\n

            If you’re about to do this<\/h3>\n
              \n
            1. Inventory every --set<\/code>.<\/strong> Each one is a future Argo CD regression.<\/li>\n
            2. Find every lookup<\/code><\/strong> in your charts. Grep the rendered templates for random-looking values that change\n between two helm template<\/code> runs.<\/li>\n
            3. Adopt with sync off<\/strong> and drive it by diff, one app at a time.<\/li>\n
            4. Seal secrets from the live cluster<\/strong> (byte-identical), not from a .env<\/code> file that may have drifted. Back up the\n sealing key somewhere that isn’t the cluster.<\/li>\n
            5. Expect to translate post-renderers and hooks<\/strong> for Argo CD.<\/li>\n
            6. Turn on self-heal last<\/strong>, after an empty diff, and test it.<\/li>\n<\/ol>\n

              Previous: Part 2 (Networking \u00b7 Next: Part 4) Upgrading with no spare capacity.<\/em><\/p>\n

              \n\n\n\n

              Get the next one in your inbox<\/h2>\n\n\n\n

              Enjoyed this? I write about local AI, self-hosting, and cloud cost ninjutsu. Join the Ninja Dispatch for occasional, high-signal notes. No fluff.<\/p>\n\n\n\n\n\t\t\t