Part 4 of the data-e2e series. Issue numbers refer to the deployment log.

The cluster came back from a two-month shutdown running Kubernetes 1.33 (end of life), Rancher 2.13 and cert-manager 1.14. Two days later it ran RKE2 1.36.5, Rancher 2.15.2 and cert-manager 1.21.2, with ingress-nginx replaced by Gateway API on the way. This post covers how that went on a cluster with no spare node, no spare RAM and a single slow disk.

The order was dictated by Rancher

Kubernetes minor versions must be upgraded one at a time (1.33 → 1.34 → 1.35 → 1.36). Rancher only supports a window of Kubernetes versions, and Rancher 2.13 stops at 1.34 (#8). The sequence was therefore:

1. RKE2 1.33 → 1.34, the most Rancher 2.13 allows.
2. Rancher 2.13 → 2.14.3 → 2.15.2, one minor version per hop.
3. cert-manager 1.14 → 1.21.2, the current release.
4. Gateway API cut-over: Traefik in, ingress-nginx out. RKE2 1.37 removes ingress-nginx, so this had to happen before going any further.
5. RKE2 1.34 → 1.35 → 1.36.

An etcd snapshot before every step, named for the step (pre-v1.36.5, pre-gateway-api, pre-rancher-2.15). On this cluster the rollback for a bad control-plane upgrade is “restore the snapshot”, so the snapshot is the upgrade’s undo button.

Upgrading nodes you can’t drain

The textbook node upgrade is cordon → drain → upgrade → uncordon. Here that fails twice over:

• There’s no room. The workers sat at 91-99 % of memory requests, so a drained node’s pods have nowhere to go.
• The pods can’t move anyway. local-path volumes are pinned to the node that holds the directory.

The opposite approach (just restart the agent on the new version) had already gone wrong once (#38). The agent took 9 minutes to come back on the busy disk. After 5 minutes NotReady, Kubernetes evicted every pod on the node.

The script that worked (rke2-upgrade-node.sh, #92) is built around keeping the restart under the 5-minute eviction window:

1. Take an etcd snapshot (server only).
2. Pre-pull the new release’s system images while the old version is still running. Image pulls were most of the downtime; doing them early takes them out of the critical path.
3. Pin settings whose defaults change in the new version. RKE2 1.36 defaults new clusters to Traefik, so ingress-controller was written explicitly into the config before upgrading.
4. Run the install script, systemctl restart --no-block, and wait for Ready.

The result: nodes were back in 357/272/182 s (1.35) and 145/123/181 s (1.36). Every one was under 300 s, and no pod was evicted. Pre-pulling turned a 10-minute risk into a 2-3 minute blip.

Rancher: upgrades that take an hour

Rancher was the slowest component by far, in ways that are worth knowing:

• --reuse-values breaks across minor versions (#82). Rancher 2.14 added a value (networkExposure) that the old release’s stored values don’t have, and the template dereferenced it: nil pointer evaluating .Values.networkExposure.type. Use --reset-then-reuse-values (Helm ≥ 3.14). It applies the new chart’s defaults and then your overrides.
• Startup probe vs. reality (#83). The new pod pulled a 913 MB image (11 min) and then restored its chart repositories for over 15 minutes. It was killed twice by its startup probe. The fix was a 60-minute startupProbe, and helm upgrade without --wait. The old pod keeps serving until the new one is Ready.
• The “jail” timeout (#86). At start Rancher copies binaries into a jail with a 60 s budget. On the HDD: [FATAL] timed out. Any Rancher setting can be overridden with an env var, so CATTLE_JAILER_TIMEOUT=900 fixed it.
• System charts that never upgraded (#91). After the upgrade, Fleet, the webhook and Turtles were still on the old chart line. Rancher had tried to install them before its chart cache loaded and never retried. Patching the rancher-charts ClusterRepo with a new forceUpdate timestamp re-ran the installs. Since then, I run helm list -A after every Rancher hop.

Gateway API cut-over without downtime

ingress-nginx and Traefik ran side by side during the switch: RKE2’s ingress-controller: [ingress-nginx, traefik], with Traefik on its own MetalLB IP and no hostPorts, so nothing clashed on 80/443.

Each app got an HTTPRoute and was tested on the new IP. DNS was flipped, and then ingress-nginx was removed by dropping it from the list. Rancher has a native Gateway mode (networkExposure.type: gateway), which replaced its Ingress with its own Gateway and routes.

One surprise: Traefik’s chart takes the Service type from service.spec.type, not service.type (#94). The Service stayed ClusterIP until that was fixed.

A repeatable upgrade, in short

1. Read the support matrix for every component that pins the Kubernetes version (Rancher, the CNI, cert-manager), and plan the hops.
2. Take an etcd snapshot per step, named after the step, and copy it off the host.
3. Pre-pull images before restarting anything. Measure node downtime and keep it under the eviction timeout.
4. Pin settings whose defaults change: read the “changed defaults” section of each release note.
5. Use --reset-then-reuse-values, never --reuse-values, across chart minor versions.
6. For slow apps, upgrade without --wait and watch the new pod; size startup probes for the worst case.
7. After the last hop, audit, not just verify: run the functional test suite (79 checks here) and look for inherited state, labels and credentials.
8. Next time: system-upgrade-controller Plans in Git instead of a script.

Previous: Part 3 (GitOps adoption · Next: Part 5). From VMs and Docker Compose to RKE2.

---

Get the next one in your inbox

Enjoyed this? I write about local AI, self-hosting, and cloud cost ninjutsu. Join the Ninja Dispatch for occasional, high-signal notes. No fluff.

No spam. Unsubscribe anytime.