{"id":3591,"date":"2026-11-04T09:00:21","date_gmt":"2026-11-04T16:00:21","guid":{"rendered":"https:\/\/yyc.ninja\/?p=3591"},"modified":"2026-11-04T09:00:21","modified_gmt":"2026-11-04T16:00:21","slug":"adopting-running-cluster-into-gitops-without-restarts","status":"publish","type":"post","link":"https:\/\/yyc.ninja\/index.php\/2026\/11\/04\/adopting-running-cluster-into-gitops-without-restarts\/","title":{"rendered":"Adopting a Running Cluster into GitOps Without Restarting a Single Pod"},"content":{"rendered":"
\"Adopting<\/figure>\n

Most Argo CD tutorials start with an empty cluster. Real platforms rarely do. The lakehouse from the storage exercise in “When Storage Is the Bottleneck” was already running 28 applications as Helm releases and raw manifests, on storage where every restart was expensive.<\/p>\n

The goal was to make Git the only way to change the cluster<\/strong>, without redeploying anything. A careless first sync could have restarted databases and reintroduced the failures that exercise had just documented, so the migration needed a strict no-restart standard.<\/p>\n

The result: 205 resources adopted, no pods replaced, and automated sync with prune and self-heal enabled at the end. Four problems had to be solved first.<\/p>\n

The approach<\/h3>\n
    \n
  1. Install Argo CD and Sealed Secrets.<\/li>\n
  2. Generate one Application<\/code> per component, using the same charts, versions and values files as the deploy scripts, and commit them to a branch Argo CD tracks.<\/li>\n
  3. Create every Application with sync disabled<\/strong>. Then, for each one:\n – review the diff;\n – change Git until the diff is empty, or shows only Argo CD’s tracking annotation;\n – sync;\n – confirm that no pod restarted;\n – move to the next application.<\/li>\n
  4. Enable automation only when every application is clean.<\/li>\n<\/ol>\n

    The diff drives the adoption at every step, so nothing depends on assumptions about what the cluster contains.<\/p>\n

    Problem 1: Secrets that existed only on the command line<\/h3>\n

    The deploy scripts passed secrets to Helm with --set<\/code>, so the values files in Git held placeholders such as password: \"set-at-install\"<\/code>. That works for scripts, but Argo CD renders only what is in Git<\/strong>. Its first sync would have replaced six real passwords with placeholders.<\/p>\n

    This risk was already proven. During an earlier manual upgrade, helm upgrade --reuse-values -f values.yaml<\/code> let the file’s placeholder override the reused secret, and JupyterHub’s password silently became set-at-install<\/code>. Nothing crashed. Logins simply started failing.<\/p>\n

    Every secret moved out of Helm values into a Kubernetes Secret that the chart references, in the style of existingSecret<\/code>. Each Secret was then sealed and committed:<\/p>\n

      \n
    • A SealedSecret is encrypted with the cluster’s public key, so it is safe in Git, and only the in-cluster controller can decrypt it.<\/li>\n
    • The sealing script copies only type<\/code> and data<\/code>, pipes directly into kubeseal<\/code>, and never writes plaintext to disk.<\/li>\n
    • 30 Secrets were sealed with byte-identical data, verified by hash before adoption.<\/li>\n
    • The deploy scripts changed from “create or update” to “create only if missing”<\/strong>, so a script run can no longer rotate a password outside Git.<\/li>\n<\/ul>\n

      Back up the sealing key.<\/strong> If the cluster is rebuilt without it, every SealedSecret in Git becomes undecryptable.<\/p>\n

      Problem 2: Values that are random by design<\/h3>\n

      Some charts generate secrets at install time with Helm’s lookup<\/code> function: reuse the existing Secret if there is one, otherwise generate a value. JupyterHub’s proxy token and cookie secret and Bitnami’s postgres-password<\/code> both work this way.<\/p>\n

      Argo CD renders charts with helm template<\/code>, which has no cluster access. lookup<\/code> always returns nothing, so the chart generates a new random value on every render<\/strong>. That creates a permanent diff and, with self-heal enabled, a password change on every sync.<\/p>\n

      The fixes, in order of preference:<\/p>\n

        \n
      1. If the chart supports an existing Secret, create one and seal it.<\/li>\n
      2. Otherwise, use ignoreDifferences<\/code> for exactly those fields, including any checksum annotation derived from them, and set RespectIgnoreDifferences=true<\/code> so a sync never writes them either.<\/li>\n<\/ol>\n

        The second option was tested directly: JupyterHub’s keys were unchanged after a sync, and the hub did not restart. The cost is that ignored fields disappear from diffs, which is acceptable for random tokens.<\/p>\n

        Problem 3: Helm features Argo CD does not have<\/h3>\n
          \n
        • Post-renderers.<\/strong> Opik needed a post-renderer to add startup probes that its MySQL sub-chart does not expose. Argo CD has no post-renderer hook, so Opik became a Kustomize<\/strong> application: helmCharts:<\/code> inflates the chart and JSON patches add the probes. The script path keeps the same patches, so the two renderings have to stay in step.<\/li>\n
        • Helm hooks.<\/strong> Argo CD honours helm.sh\/hook<\/code> only for Helm sources. A chart inflated through Kustomize turned its MinIO bucket job into an ordinary resource, which went missing once its TTL deleted it and left the application permanently OutOfSync. Argo CD’s own annotation, argocd.argoproj.io\/hook: PostSync<\/code>, fixed this, and also worked for charts with TTL-managed migration Jobs such as Airflow.<\/li>\n
        • Large CRDs.<\/strong> The Spark Operator’s CRD exceeds the 256 KB annotation limit that client-side apply uses for last-applied-configuration<\/code>. Setting ServerSideApply=true<\/code> for that application, and for Strimzi, KServe and Opik, avoids the annotation entirely.<\/li>\n<\/ul>\n

          Problem 4: IPv6 on an IPv4-only pod network<\/h3>\n

          The repo-server could not fetch the repository, failing with git fetch ... timeout after 1m40s<\/code>. The pods have IPv4 only, and the Git host answered on IPv6 first. An SSH configuration with AddressFamily inet<\/code> fixed it. The networking article in this series covers the root cause.<\/p>\n

          The adoption<\/h3>\n

          With those problems solved, the diffs across all 205 resources were uneventful. The only differences were Argo CD’s tracking annotation and one Helm label. Each application was synced individually, with kubectl get pods<\/code> checked for restarts after each one.<\/p>\n

          Automation then went on for all 28 applications: automated: {prune: true, selfHeal: true}<\/code>.<\/p>\n

          The final test was deliberate drift. A Deployment changed by hand was reverted by Argo CD in 10 seconds<\/strong>. From that point, kubectl edit<\/code> is only a temporary experiment, and Git is the write path.<\/p>\n

          What changed in daily operations<\/h3>\n
            \n
          • A change is a commit.<\/strong> Values, manifests and Sealed Secrets go to main<\/code>, and Argo CD applies them within 5 minutes. It polls, because the cluster does not accept inbound webhooks from the Git host.<\/li>\n
          • The scripts remain for bootstrap and break-glass use.<\/strong> They never overwrite existing Secrets, so they cannot conflict with Argo CD.<\/li>\n
          • Drift is visible.<\/strong> OutOfSync now means someone changed the cluster outside Git, and with self-heal that change has already been undone.<\/li>\n<\/ul>\n

            Recommendations<\/h3>\n
              \n
            1. Inventory every --set<\/code>.<\/strong> Each one is a future Argo CD regression.<\/li>\n
            2. Find every lookup<\/code> in your charts<\/strong> by comparing two helm template<\/code> runs for values that change.<\/li>\n
            3. Adopt with sync off<\/strong> and work through the diffs one application at a time.<\/li>\n
            4. Seal secrets from the live cluster,<\/strong> not from a .env<\/code> file that may have drifted, and keep a backup of the sealing key outside the cluster.<\/li>\n
            5. Plan to translate post-renderers and hooks<\/strong> for Argo CD.<\/li>\n
            6. Enable self-heal last,<\/strong> after an empty diff, and test it.<\/li>\n<\/ol>\n\n\n
              \n\n\n\n

              Get the next one in your inbox<\/h2>\n\n\n\n

              Enjoyed this? I write about local AI, self-hosting, and cloud cost ninjutsu. Join the Ninja Dispatch for occasional, high-signal notes. No fluff.<\/p>\n\n\n\n\t\t\t