Part 2 of the data-e2e series. Issue numbers (#95) refer to the deployment log.

Getting 13 web UIs out of a Kubernetes cluster, onto proper HTTPS names that work from every device in the house, sounds like a solved problem. It mostly is. This post is about the parts that weren’t: the bits where a home network behaves nothing like a cloud VPC.

The shape of it

browser ─► LAN DNS (Technitium): *.apps.example.com → <lan>.128
        ─► Nginx Proxy Manager (.128, certificate *.apps.example.com)
        ─► Traefik on MetalLB IP <lan>.241 (SNI = the requested host)
        ─► Gateway "lakehouse" ─► HTTPRoute per app ─► ClusterIP Service

Each layer was chosen for a reason:

• Gateway API instead of Ingress. ingress-nginx was retired upstream in March 2026, and RKE2 1.36 makes Traefik the default. Gateway API also splits responsibilities: one shared Gateway (owned by the platform) holds the wildcard certificate, and each namespace attaches its own HTTPRoute. A namespace opts in with a label (gateway.example.com/routes=true), so a tenant can’t hijack someone else’s hostname.
• MetalLB gives Traefik a real LAN IP (.241) in L2 mode: one node answers ARP for it. There are no cloud load balancers at home.
• One wildcard certificate from Let’s Encrypt via DNS-01 (cert-manager + a Cloudflare API token), so no app needs its own certificate and nothing has to be reachable from the internet to get one.
• Nginx Proxy Manager in front. The house already had two NPMs (one per domain family). Sending all LAN and, later, Cloudflare-tunnel traffic through the one that owns apps.example.com means there is one entry point to secure. A new app needs only an HTTPRoute: no DNS record, no NPM change, thanks to a wildcard at both layers (#112).

Now the traps.

Trap 1: the LAN answers for the whole internet (DNS-01 stuck)

The certificate wouldn’t issue: “DNS record not yet propagated”, though the TXT record was plainly visible in the Cloudflare dashboard (#95). Querying Cloudflare’s authoritative nameservers directly from inside the LAN returned NXDOMAIN.

The tell was in the flags. The answer had rd ra and no aa, so it was not from an authoritative server at all. The home network intercepts every DNS packet on port 53, whatever IP it’s addressed to (common with parental controls and some routers). The local resolver had a forwarder zone for apps.example.com and knew nothing about _acme-challenge.

Fix: let cert-manager check propagation over DNS-over-HTTPS, which can’t be intercepted that way:

--dns01-recursive-nameservers-only
--dns01-recursive-nameservers=https://cloudflare-dns.com/dns-query,https://dns.google/dns-query

The certificate issued in two minutes and renews automatically.

Trap 2: the router throws away private answers (rebind protection)

Everything worked from the workstation and nothing worked from the phones (#110).

• The workstation asks Technitium directly.
• Phones ask the router, which asks Technitium.
• The router runs dnsmasq with rebind_protection: if a public domain answers with a private IP (10.x, 192.168.x), dnsmasq assumes a DNS-rebinding attack and replaces the answer with nothing. The client sees NOERROR with zero records and gives up.

The fix is one line per internal domain (rebind_domain='<your-domain>'), plus the same line in the router’s restore-after-flash script. Without the second part, the next firmware upgrade silently breaks it again, which is exactly what happened here once already.

This one is worth remembering when you add a Cloudflare tunnel. From the internet everything works, because public DNS points at Cloudflare. From home, the same names fail until the rebind whitelist is in place. “Works outside, broken inside” is the signature.

Trap 3: a forwarder pointing at itself

Technitium’s apps.example.com zone was a forwarder zone whose forwarder was 127.0.0.1, itself (#111). Any name without a local record (the apex, www) returned SERVFAIL locally. The fix: real upstream forwarders plus one wildcard record *.apps.example.com → NPM.

An attempted apex record pointing at NPM was rolled back. NPM had no proxy host for the apex, so LAN clients lost the public website. Wildcards don’t cover the apex, and that’s usually what you want.

Trap 4: IPv6 that isn’t there

Argo CD couldn’t fetch a 180 KB Git repository: git fetch … timeout after 1m40s (#100). The pods have IPv4 only, but gitlab.com resolves to an IPv6 address first. OpenSSH tried it, waited on a connect that could never succeed, and Argo’s 90-second exec timeout won the race.

Fix: a mounted SSH client config with AddressFamily inet. It’s a good default for any homelab pod network that lacks IPv6.

The bug that looked like a broken login form

About half of all JupyterHub logins through the Gateway failed with “Login form invalid or expired” (#108). The other half worked. Same browser, same password.

• JupyterHub’s anonymous login token (XSRF) is bound to the client IP and user agent.
• Traefik’s Service used externalTrafficPolicy: Cluster. MetalLB delivers traffic to one node, and kube-proxy may forward it to a Traefik pod on another node, rewriting the source IP (SNAT) to that node’s address on the way.
• So the GET that rendered the form and the POST that submitted it reached JupyterHub from two different “client IPs”.

Fix: externalTrafficPolicy: Local. Traffic now enters only on the node running Traefik, the real client IP is preserved, and logins went 10/10. The verification script now logs in three times in a row. A single success proves nothing for an intermittent bug.

Apps that don’t trust proxies

Putting apps behind a reverse proxy surfaces each app’s opinion about X-Forwarded-* and Host (#96, #99):

• Trino returned 406 until http-server.process-forwarded=true.
• MLflow 3 has a host-header allow-list (serverAllowedHosts) and returned 403 for the new name.
• NiFi 2 enforces SNI against its own certificate. That certificate is generated at first start from the proxy hosts it knows then. Adding a name later means regenerating the keystore. The documented “disable SNI check” switch has no effect in 2.12.

None of these are bugs. They are security features doing their job, and they all fail closed. Expect one per app.

NetworkPolicies cut both ways

JupyterHub’s chart ships a NetworkPolicy that blocks notebook egress to private IPs. In a cluster, that’s every Service (#71). From a notebook, MLflow, Trino and S3 all timed out.

The tempting fix (privateIPs: true) would also have opened the entire home LAN to every notebook. Instead the policy allows egress to the data-platform namespaces only, so each new namespace has to be added deliberately. Default-deny with explicit allows is the model worth following everywhere.

Checklist

1. Choose the LB range and remove it from DHCP: pool and reservations.
2. Whitelist every split-horizon domain in the router’s rebind protection, and in its restore script.
3. Use DoH (or a resolver you control) for ACME DNS-01 checks if anything on the LAN intercepts port 53.
4. externalTrafficPolicy: Local on the ingress Service if any app ties sessions to client IPs, or if you want real IPs in your logs.
5. Force IPv4 for SSH and Git in pods if the cluster has no IPv6.
6. Per app: forwarded headers, allowed hosts, SNI.
7. Wildcard DNS + wildcard certificate + one Gateway: adding an app should be one HTTPRoute and nothing else.

Previous: Part 1 (one spinning disk · Next: Part 3) Adopting a live cluster into GitOps.

---

Get the next one in your inbox

Enjoyed this? I write about local AI, self-hosting, and cloud cost ninjutsu. Join the Ninja Dispatch for occasional, high-signal notes. No fluff.

No spam. Unsubscribe anytime.