Rolling upgrades assume spare capacity: drain a node, let its workloads reschedule, upgrade, repeat. This test platform was deliberately built without that margin, so it was a good place to find out what a safe upgrade looks like when the standard procedure cannot work.

The cluster started on Kubernetes 1.33 (end of life), Rancher 2.13 and cert-manager 1.14. Two days later it ran RKE2 1.36.5, Rancher 2.15.2 and cert-manager 1.21.2, with ingress-nginx replaced by Gateway API along the way. All of this happened with no spare node, no spare memory and the slow shared storage described in “When Storage Is the Bottleneck”.

Rancher’s support matrix set the order

Kubernetes minor versions must be upgraded one at a time. Rancher supports only a window of Kubernetes versions, and Rancher 2.13 stops at 1.34. That dictated the sequence:

1. RKE2 1.33 to 1.34, the highest version Rancher 2.13 allows.
2. Rancher 2.13 to 2.14.3 to 2.15.2, one minor version per hop.
3. cert-manager 1.14 to 1.21.2, the current release.
4. Gateway API cut-over, bringing in Traefik and removing ingress-nginx. RKE2 1.37 removes ingress-nginx, so this had to happen before going further.
5. RKE2 1.34 to 1.35 to 1.36.

An etcd snapshot preceded every step, named after that step (pre-v1.36.5, pre-gateway-api, pre-rancher-2.15). The rollback for a bad control plane upgrade is to restore the snapshot, so each snapshot is that step’s undo button.

Upgrading nodes that cannot be drained

The textbook sequence is cordon, drain, upgrade, uncordon. It fails here for two reasons:

• There is no room. Workers sat at 91% to 99% of memory requests, so a drained node’s pods had nowhere to go.
• The pods cannot move. local-path volumes are pinned to the node that holds the directory.

The opposite approach, simply restarting the agent on the new version, had already failed. The agent took 9 minutes to return on the busy disk, and after 5 minutes in NotReady, Kubernetes evicted every pod on the node.

The upgrade script that worked was designed around one constraint: keep the restart inside the 5-minute eviction window.

1. Take an etcd snapshot (server nodes only).
2. Pre-pull the new release’s system images while the old version is still running. Image pulls were most of the downtime, so doing them early removes them from the critical path.
3. Pin settings whose defaults change in the new version. RKE2 1.36 defaults new clusters to Traefik, so ingress-controller was set explicitly before upgrading.
4. Run the install script, use systemctl restart --no-block, and wait for Ready.

Nodes returned in 357, 272 and 182 seconds on the 1.35 upgrade, and in 145, 123 and 181 seconds on 1.36. No pod was evicted. Pre-pulling cut a 9-minute restart to about 2 to 3 minutes by the final hop.

Rancher: upgrades that take an hour

Rancher was the slowest component by a wide margin:

• --reuse-values breaks across minor versions. Rancher 2.14 added a value (networkExposure) that the old release’s stored values lack, and the template dereferenced it: nil pointer evaluating .Values.networkExposure.type. Use --reset-then-reuse-values (Helm 3.14 or later), which applies the new chart’s defaults and then your overrides.
• Startup probes need the worst-case budget. The new pod pulled a 913 MB image in 11 minutes, then restored its chart repositories for more than 15 minutes, and its startup probe killed it twice. The fix was a 60-minute startupProbe and helm upgrade without --wait. The old pod keeps serving until the new one is Ready.
• The jail timeout. At startup, Rancher copies binaries into a jail with a 60-second budget, which failed on this storage with [FATAL] timed out. Rancher settings can be overridden through environment variables, so CATTLE_JAILER_TIMEOUT=900 solved it.
• System charts that never upgraded. After the upgrade, Fleet, the webhook and Turtles were still on the old chart line. Rancher had tried to install them before its chart cache loaded and never retried. Patching the rancher-charts ClusterRepo with a new forceUpdate timestamp re-ran the installs. helm list -A is now a required check after every Rancher hop.

Gateway API cut-over without downtime

ingress-nginx and Traefik ran side by side during the switch, using RKE2’s ingress-controller: [ingress-nginx, traefik]. Traefik had its own MetalLB address and no hostPorts, so nothing conflicted on ports 80 and 443.

Each application received an HTTPRoute and was tested on the new address. DNS was then switched, and ingress-nginx was removed by dropping it from the list. Rancher’s native Gateway mode (networkExposure.type: gateway) replaced its Ingress with its own Gateway and routes.

One detail to watch: Traefik’s chart takes the Service type from service.spec.type, not service.type. The Service stayed ClusterIP until this was corrected.

A repeatable upgrade process

1. Read the support matrix for every component that pins the Kubernetes version, including Rancher, the CNI and cert-manager, and plan the hops.
2. Take one etcd snapshot per step, named after the step, and copy it off the node.
3. Pre-pull images before restarting anything, and measure node downtime against the eviction timeout.
4. Pin settings whose defaults change, using the changed-defaults section of each release note.
5. Use --reset-then-reuse-values, not --reuse-values, across chart minor versions.
6. For slow applications, upgrade without --wait, watch the new pod, and size startup probes for the worst case.
7. After the last hop, audit as well as verify: run the functional test suite (79 checks here) and look for inherited state, labels and credentials.
8. As a next improvement, replace the script with system-upgrade-controller Plans in Git.

---

Get the next one in your inbox

Enjoyed this? I write about local AI, self-hosting, and cloud cost ninjutsu. Join the Ninja Dispatch for occasional, high-signal notes. No fluff.

No spam. Unsubscribe anytime.