Part 5 of the data-e2e series. The concrete plan for this homelab is the project’s planning docs; the phase method is the project’s planning docs §18.

A typical homelab grows the same way. A VM gets Docker Compose, then a second stack, then a third VM “for AI things”. After a few years:

• one VM runs an LLM gateway, a search engine and a chat UI;
• another runs n8n, an RSS reader and an 11-container Dify;
• a third runs Windmill with workers that drive the host’s Docker socket.

Every service has a LAN IP and port that something, somewhere, has hard-coded.

Meanwhile a Kubernetes cluster now exists for the lakehouse (Parts 1-4), with GitOps, a Gateway, MetalLB and backups. The question is how to move the VM workloads onto it so the VMs can actually be switched off, without editing every client that knows <lan>.82:4000.

Principle 1: move the IP, not the clients

The instinct is to give each app a nice new hostname and update the clients. Resist it for the first move. Clients are where migrations die: a webhook registered with a third party, a URL in a Windmill script, a Secret in another namespace, a bookmark on a phone.

MetalLB can take over the old VM’s IP.

1. Create a second address pool, preserved-service-ips, with autoAssign: false so nothing lands there by accident. It starts empty.
2. At cut-over, after the VM is shut down, add the VM’s address as a /32.
3. Create one LoadBalancer Service per app with loadBalancerIPs: <old ip> and the old port.
4. When several apps lived on one VM, their Services share the IP. Give them the same metallb.universe.tf/allow-shared-ip key and distinct ports.

MetalLB announces the address with a gratuitous ARP, and within seconds <lan>.82:4000 is LiteLLM-in-Kubernetes. Every reverse-proxy host, DNS record and hard-coded client keeps working without an edit. Hostnames can come later, at leisure, through the Gateway.

Three rules make this safe:

• Never take an IP that is still alive. MetalLB will happily ARP for an address another machine is using. qm shutdown first, then add the /32.
• Keep the router’s DHCP reservation, renamed (metallb-llm-mcp). The reservation binds the address to the dead VM’s MAC, so DHCP can never hand it to a laptop. Delete it and another device will get .82 next week.
• Keep the load-balancer range out of the DHCP pool, so no dynamic lease can ever land on an address MetalLB announces.

Principle 2: translate Docker’s semantics, don’t copy them

docker ps and docker inspect tell you more than the Compose file, because they show what is actually running. The mapping:

On the VM	In Kubernetes
restart: always / unless-stopped	Deployment or StatefulSet + liveness/readiness probes + a startupProbe sized for your disks
Container stopped by hand (unless-stopped, exited)	replicas: 0 in Git, or don’t migrate it. Argo CD’s self-heal keeps it at 0
restart: no one-shot (e.g. init_permissions)	initContainer or a hook Job
Container in a restart loop	Usually don’t migrate. Find out why first; often it’s a sidecar the cluster already provides (cAdvisor vs. the cluster’s metrics)
Named volume	PVC. Databases on the fast tier, bulk files on the big one
Bind mount of a config file	ConfigMap (secrets → SealedSecret)
ports: "0.0.0.0:5432:5432"	Probably nothing. Database ports published to the whole LAN are usually accidents. ClusterIP only, unless a real client shows up in a week of connection logs
image: …:latest	The digest the VM is running now (docker inspect -f '{{.Image}}'). Upgrade after the migration has soaked, never during it
Host systemd unit next to Docker	Containerise it, or that VM can’t be decommissioned yet
Container with /var/run/docker.sock	A red flag. The app drives Docker itself (Windmill workers, CI runners, Portainer). Needs a redesign (native or Kubernetes workers), not a lift-and-shift

In this homelab that last row disqualified a whole VM from the first wave. The other two VMs are almost entirely “one container, one volume, one port”, which is the shape that migrates cleanly.

Principle 3: one VM per phase, with the same seven steps

Each phase runs gate → back up → build → validate → cut over → soak → decommission.

1. Gate. Check host RAM, swap, I/O wait and backup-datastore usage against written thresholds. A migration that starts on a stressed host fails for reasons unrelated to the migration.
2. Back up. – A protected PBS backup of the VM. Keep the qm config dump with it; a VM restore without its definition is not a restore. – Application-level exports that don’t depend on the VM: pg_dump, n8n’s volume including its encryption key, a tar of the chat UI’s data directory.
3. Build the new version on a temporary MetalLB IP, from Git, with the data restored from the exports.
4. Validate with golden checks captured from the old service first: – the same model list from the LLM gateway, and one completion per model; – the same workflow count in n8n; – the same top-5 results for 20 fixed search queries.

Scripted, not eyeballed. 5. Cut over. – Stop the containers, then qm shutdown. – Add the /32 and move the Services from the temporary IP to the old one. – Run the golden checks again. – Rollback is the reverse and takes minutes: remove the /32, qm start. 6. Soak 7 days: scheduled jobs run on time, there are no duplicate notifications, and backups of the new state exist. 7. Decommission. The VM stays stopped (not destroyed) for another week, and its backup stays protected for 30 days. Then qm destroy.

The trap specific to automation tools: running twice

n8n, changedetection and anything else that acts (sends mail, calls webhooks, posts messages) will act twice if both copies run. Import every workflow into the new instance inactive, validate with manual executions, and activate only after the old instance is stopped. The old instance comes back only through an explicit rollback.

Don’t forget the things around the app

• Reverse proxies and DNS. With IP takeover they need no change. List them anyway, so you know who to check. Here two NPM instances pointed flow.home.example.net and fresh.home.example.net at .114.
• Tailscale and other overlay names. A VM that is a tailnet node takes its MagicDNS name with it when it dies. Remove the node first, then give the new Service the same name with the Tailscale Kubernetes operator, or advertise the /32 from a subnet router.
• Monitoring. The VM’s own agents (Alloy, cAdvisor, node-exporter) disappear and the cluster’s take over. Update the blackbox and alert targets, or the dead VM shows as DOWN forever.
• Dead proxy hosts. The inventory also finds entries pointing at ports where nothing listens any more. Delete them now; they confuse the next migration.

Choosing what goes first

For each VM, ask four questions:

• How much state does it hold, and is there a clean export?
• How many clients hard-code it?
• Does it do anything Kubernetes can’t (Docker socket, host services, inbound SMTP)?
• How much RAM does it free?

Here that produced:

1. The LLM gateway VM. Little state (one Postgres plus config), the highest fan-out (the lakehouse itself calls it), and it frees 8 GB.
2. The automation VM. Simple apps plus one big one (Dify) that must move before the VM can go.
3. The search VM. Already stopped, so there’s no live cut-over, but it adds RAM and needs a new node. That is what the golden-image template is for: add-rke2-node.sh clones a pre-hardened node and joins it to the cluster in about 5 minutes.

The VM with Docker-socket workers, SMTP and a WhatsApp session waits. Moving it is a redesign, not a migration, and it should get that time on its own.

The payoff

A VM you migrate this way leaves behind:

• an IP that still answers, served by the cluster;
• a Git history of every config change;
• a Sealed Secret instead of an .env file on a disk nobody backs up;
• probes that restart it when it hangs;
• 8-16 GB of host RAM returned.

And, finally, a VM you can switch off.

Previous: Part 4 (Upgrades · Start: Part 1), one spinning disk.

---

Get the next one in your inbox

Enjoyed this? I write about local AI, self-hosting, and cloud cost ninjutsu. Join the Ninja Dispatch for occasional, high-signal notes. No fluff.

No spam. Unsubscribe anytime.