{"id":3529,"date":"2026-06-09T10:26:00","date_gmt":"2026-06-09T16:26:00","guid":{"rendered":"https:\/\/yyc.ninja\/index.php\/2026\/10\/20\/upgrading-rke2-rancher-cert-manager-with-no-spare-capacity\/"},"modified":"2026-10-20T13:00:21","modified_gmt":"2026-10-20T19:00:21","slug":"upgrading-rke2-rancher-cert-manager-with-no-spare-capacity","status":"publish","type":"post","link":"https:\/\/yyc.ninja\/index.php\/2026\/06\/09\/upgrading-rke2-rancher-cert-manager-with-no-spare-capacity\/","title":{"rendered":"Upgrading Kubernetes, Rancher and cert-manager with no spare capacity"},"content":{"rendered":"
\"Upgrading<\/figure>\n

Part 4 of the data-e2e series. Issue numbers refer to the deployment log.<\/em><\/p>\n

The cluster came back from a two-month shutdown running Kubernetes 1.33<\/strong> (end of life), Rancher 2.13<\/strong> and\ncert-manager 1.14<\/strong>. Two days later it ran RKE2 1.36.5, Rancher 2.15.2 and cert-manager 1.21.2<\/strong>, with\ningress-nginx replaced by Gateway API on the way. This post covers how that went on a cluster with no spare node,\nno spare RAM and a single slow disk.<\/p>\n

The order was dictated by Rancher<\/h3>\n

Kubernetes minor versions must be upgraded one at a time (1.33 \u2192 1.34 \u2192 1.35 \u2192 1.36). Rancher only supports a\nwindow of Kubernetes versions, and Rancher 2.13 stops at 1.34<\/strong> (#8). The sequence was therefore:<\/p>\n

    \n
  1. RKE2 1.33 \u2192 1.34, the most Rancher 2.13 allows.<\/li>\n
  2. Rancher 2.13 \u2192 2.14.3 \u2192 2.15.2, one minor version per hop.<\/li>\n
  3. cert-manager 1.14 \u2192 1.21.2, the current release.<\/li>\n
  4. Gateway API cut-over: Traefik in, ingress-nginx out. RKE2 1.37 removes ingress-nginx, so this had to happen\n before going any further.<\/li>\n
  5. RKE2 1.34 \u2192 1.35 \u2192 1.36.<\/li>\n<\/ol>\n

    An etcd snapshot before every step<\/strong>, named for the step (pre-v1.36.5<\/code>, pre-gateway-api<\/code>, pre-rancher-2.15<\/code>).\nOn this cluster the rollback for a bad control-plane upgrade is “restore the snapshot”, so the snapshot is the\nupgrade’s undo button.<\/p>\n

    Upgrading nodes you can’t drain<\/h3>\n

    The textbook node upgrade is cordon \u2192 drain \u2192 upgrade \u2192 uncordon. Here that fails twice over:<\/p>\n

      \n
    • There’s no room.<\/strong> The workers sat at 91-99 % of memory requests, so a drained node’s pods have nowhere to go.<\/li>\n
    • The pods can’t move anyway.<\/strong> local-path volumes are pinned to the node that holds the directory.<\/li>\n<\/ul>\n

      The opposite approach (just restart the agent on the new version) had already gone wrong once (#38). The agent\ntook 9 minutes to come back on the busy disk. After 5 minutes NotReady, Kubernetes evicted every pod on the node.<\/p>\n

      The script that worked (rke2-upgrade-node.sh<\/code>, #92) is built around keeping the restart under the 5-minute\neviction window<\/strong>:<\/p>\n

        \n
      1. Take an etcd snapshot (server only).<\/li>\n
      2. Pre-pull the new release’s system images while the old version is still running.<\/strong> Image pulls were most of\n the downtime; doing them early takes them out of the critical path.<\/li>\n
      3. Pin settings whose defaults<\/em> change in the new version. RKE2 1.36 defaults new clusters to Traefik, so\n ingress-controller<\/code> was written explicitly into the config before upgrading.<\/li>\n
      4. Run the install script, systemctl restart --no-block<\/code>, and wait for Ready.<\/li>\n<\/ol>\n

        The result: nodes were back in 357\/272\/182 s<\/strong> (1.35) and 145\/123\/181 s<\/strong> (1.36). Every one was under 300 s, and\nno pod was evicted. Pre-pulling turned a 10-minute risk into a 2-3 minute blip.<\/p>\n

        Rancher: upgrades that take an hour<\/h3>\n

        Rancher was the slowest component by far, in ways that are worth knowing:<\/p>\n

          \n
        • --reuse-values<\/code> breaks across minor versions (#82).<\/strong> Rancher 2.14 added a value (networkExposure<\/code>) that the\n old release’s stored values don’t have, and the template dereferenced it: nil pointer evaluating\n .Values.networkExposure.type<\/code>. Use --reset-then-reuse-values<\/code> (Helm \u2265 3.14). It applies the new<\/em> chart’s\n defaults and then your overrides.<\/li>\n
        • Startup probe vs. reality (#83).<\/strong> The new pod pulled a 913 MB image (11 min) and then restored its chart\n repositories for over 15 minutes. It was killed twice by its startup probe. The fix was a 60-minute\n startupProbe<\/code>, and helm upgrade<\/code> without<\/em> --wait<\/code>. The old pod keeps serving until the new one is Ready.<\/li>\n
        • The “jail” timeout (#86).<\/strong> At start Rancher copies binaries into a jail with a 60 s budget. On the HDD: [FATAL]\n timed out<\/code>. Any Rancher setting can be overridden with an env var, so CATTLE_JAILER_TIMEOUT=900<\/code> fixed it.<\/li>\n
        • System charts that never upgraded (#91).<\/strong> After the upgrade, Fleet, the webhook and Turtles were still on the old\n chart line. Rancher had tried to install them before its chart cache loaded and never retried. Patching the\n rancher-charts<\/code> ClusterRepo with a new forceUpdate<\/code> timestamp re-ran the installs. Since then, I run\n helm list -A<\/code> after every Rancher hop.<\/li>\n<\/ul>\n

          Gateway API cut-over without downtime<\/h3>\n

          ingress-nginx and Traefik ran side by side<\/strong> during the switch: RKE2’s ingress-controller: [ingress-nginx,\ntraefik]<\/code>, with Traefik on its own MetalLB IP and no hostPorts, so nothing clashed on 80\/443.<\/p>\n

          Each app got an HTTPRoute and was tested on the new IP. DNS was flipped, and then ingress-nginx was removed by\ndropping it from the list. Rancher has a native Gateway mode (networkExposure.type: gateway<\/code>), which replaced its\nIngress with its own Gateway and routes.<\/p>\n

          One surprise: Traefik’s chart takes the Service type from service.spec.type<\/code>, not service.type<\/code> (#94). The Service\nstayed ClusterIP until that was fixed.<\/p>\n

          A repeatable upgrade, in short<\/h3>\n
            \n
          1. Read the support matrix for every<\/strong> component that pins the Kubernetes version (Rancher, the CNI,\n cert-manager), and plan the hops.<\/li>\n
          2. Take an etcd snapshot per step<\/strong>, named after the step, and copy it off the host.<\/li>\n
          3. Pre-pull images<\/strong> before restarting anything. Measure node downtime and keep it under the eviction timeout.<\/li>\n
          4. Pin settings whose defaults change: read the “changed defaults” section of each release note.<\/li>\n
          5. Use --reset-then-reuse-values<\/code>, never --reuse-values<\/code>, across chart minor versions.<\/li>\n
          6. For slow apps, upgrade without --wait<\/code> and watch the new pod; size startup probes for the worst case.<\/li>\n
          7. After the last hop, audit, not just verify<\/strong>: run the functional test suite (79 checks here) and<\/em> look for\n inherited state, labels and credentials.<\/li>\n
          8. Next time: system-upgrade-controller Plans in Git instead of a script.<\/li>\n<\/ol>\n

            Previous: Part 3 (GitOps adoption \u00b7 Next: Part 5). From VMs and Docker Compose to\nRKE2.<\/em><\/p>\n\n\n

            \n\n\n\n

            Get the next one in your inbox<\/h2>\n\n\n\n

            Enjoyed this? I write about local AI, self-hosting, and cloud cost ninjutsu. Join the Ninja Dispatch for occasional, high-signal notes. No fluff.<\/p>\n\n\n\n\t\t\t