{"id":3527,"date":"2026-05-26T09:38:00","date_gmt":"2026-05-26T15:38:00","guid":{"rendered":"https:\/\/yyc.ninja\/index.php\/2026\/10\/20\/networking-a-homelab-kubernetes-cluster-gateway-api-metallb-dns-traps\/"},"modified":"2026-10-23T11:26:19","modified_gmt":"2026-10-23T17:26:19","slug":"networking-a-homelab-kubernetes-cluster-gateway-api-metallb-dns-traps","status":"publish","type":"post","link":"https:\/\/yyc.ninja\/index.php\/2026\/05\/26\/networking-a-homelab-kubernetes-cluster-gateway-api-metallb-dns-traps\/","title":{"rendered":"Networking a homelab Kubernetes cluster: Gateway API, MetalLB and four DNS traps"},"content":{"rendered":"
\"Networking<\/figure>\n

Part 2 of the data-e2e series. Issue numbers (#95) refer to the deployment log.<\/em><\/p>\n

Getting 13 web UIs out of a Kubernetes cluster, onto proper HTTPS names that work from every device in the house,\nsounds like a solved problem. It mostly is. This post is about the parts that weren’t: the bits where a home network\nbehaves nothing like a cloud VPC.<\/p>\n

The shape of it<\/h3>\n
browser \u2500\u25ba LAN DNS (Technitium): *.apps.example.com \u2192 <lan>.128\n        \u2500\u25ba Nginx Proxy Manager (.128, certificate *.apps.example.com)\n        \u2500\u25ba Traefik on MetalLB IP <lan>.241 (SNI = the requested host)\n        \u2500\u25ba Gateway "lakehouse" \u2500\u25ba HTTPRoute per app \u2500\u25ba ClusterIP Service\n<\/code><\/pre>\n
Each layer was chosen for a reason:<\/p>\n
    \n
  • Gateway API instead of Ingress.<\/strong> ingress-nginx was retired upstream in March 2026, and RKE2 1.36 makes Traefik\n the default. Gateway API also splits responsibilities: one shared Gateway<\/code> (owned by the platform) holds the\n wildcard certificate, and each namespace attaches its own HTTPRoute<\/code>. A namespace opts in with a label\n (gateway.example.com\/routes=true<\/code>), so a tenant can’t hijack someone else’s hostname.<\/li>\n
  • MetalLB<\/strong> gives Traefik a real LAN IP (.241<\/code>) in L2 mode: one node answers ARP for it. There are no cloud load\n balancers at home.<\/li>\n
  • One wildcard certificate<\/strong> from Let’s Encrypt via DNS-01 (cert-manager + a Cloudflare API token), so no app needs\n its own certificate and nothing has to be reachable from the internet to get one.<\/li>\n
  • Nginx Proxy Manager in front.<\/strong> The house already had two NPMs (one per domain family). Sending all LAN and,\n later, Cloudflare-tunnel traffic through the one that owns apps.example.com<\/code> means there is one entry point to secure.\n A new app needs only an HTTPRoute: no DNS record, no NPM change, thanks to a wildcard at both layers (#112).<\/li>\n<\/ul>\n
    Now the traps.<\/p>\n
    Trap 1: the LAN answers for the whole internet (DNS-01 stuck)<\/h3>\n
    The certificate wouldn’t issue: “DNS record not yet propagated”, though the TXT record was plainly visible in the\nCloudflare dashboard (#95). Querying Cloudflare’s authoritative nameservers directly<\/em> from inside the LAN returned\nNXDOMAIN.<\/p>\n
    The tell was in the flags. The answer had rd ra<\/code> and no aa<\/code>, so it was not from an authoritative server at all.\nThe home network intercepts every DNS packet on port 53<\/strong>, whatever IP it’s addressed to (common with parental\ncontrols and some routers). The local resolver had a forwarder zone for apps.example.com<\/code> and knew nothing about\n_acme-challenge<\/code>.<\/p>\n
    Fix:<\/strong> let cert-manager check propagation over DNS-over-HTTPS, which can’t be intercepted that way:<\/p>\n
    --dns01-recursive-nameservers-only\n--dns01-recursive-nameservers=https:\/\/cloudflare-dns.com\/dns-query,https:\/\/dns.google\/dns-query\n<\/code><\/pre>\n
    The certificate issued in two minutes and renews automatically.<\/p>\n
    Trap 2: the router throws away private answers (rebind protection)<\/h3>\n
    Everything worked from the workstation and nothing worked from the phones (#110).<\/p>\n
      \n
    • The workstation asks Technitium directly.<\/li>\n
    • Phones ask the router, which asks Technitium.<\/li>\n
    • The router runs dnsmasq with rebind_protection<\/code>: if a public domain answers with a private IP (10.x<\/code>,\n 192.168.x<\/code>), dnsmasq assumes a DNS-rebinding attack and replaces the answer with nothing<\/em>. The client sees NOERROR\n with zero records and gives up.<\/li>\n<\/ul>\n
      The fix is one line per internal domain (rebind_domain='<your-domain>'<\/code>), plus the same line in the router’s\nrestore-after-flash script. Without the second part, the next firmware upgrade silently breaks it again, which is\nexactly what happened here once already.<\/p>\n
      This one is worth remembering when you add a Cloudflare tunnel<\/strong>. From the internet everything works, because\npublic DNS points at Cloudflare. From home, the same names fail until the rebind whitelist is in place. “Works\noutside, broken inside” is the signature.<\/p>\n
      Trap 3: a forwarder pointing at itself<\/h3>\n
      Technitium’s apps.example.com<\/code> zone was a forwarder<\/em> zone whose forwarder was 127.0.0.1<\/code>, itself (#111). Any name without\na local record (the apex, www<\/code>) returned SERVFAIL locally. The fix: real upstream forwarders plus one wildcard record\n*.apps.example.com \u2192 NPM<\/code>.<\/p>\n
      An attempted apex record pointing at NPM was rolled back. NPM had no proxy host for the apex, so LAN clients lost the\npublic website. Wildcards don’t cover the apex, and that’s usually what you want.<\/p>\n
      Trap 4: IPv6 that isn’t there<\/h3>\n
      Argo CD couldn’t fetch a 180 KB Git repository: git fetch \u2026 timeout after 1m40s<\/code> (#100). The pods have IPv4 only,\nbut gitlab.com<\/code> resolves to an IPv6 address first. OpenSSH tried it, waited on a connect that could never succeed,\nand Argo’s 90-second exec timeout won the race.<\/p>\n
      Fix:<\/strong> a mounted SSH client config with AddressFamily inet<\/code>. It’s a good default for any homelab pod network that\nlacks IPv6.<\/p>\n
      The bug that looked like a broken login form<\/h3>\n
      About half of all JupyterHub logins through the Gateway failed with “Login form invalid or expired” (#108). The other\nhalf worked. Same browser, same password.<\/p>\n
        \n
      • JupyterHub’s anonymous login token (XSRF) is bound to the client IP<\/strong> and user agent.<\/li>\n
      • Traefik’s Service used externalTrafficPolicy: Cluster<\/code>. MetalLB delivers traffic to one node, and kube-proxy may\n forward it to a Traefik pod on another<\/em> node, rewriting the source IP (SNAT) to that node’s address on the way.<\/li>\n
      • So the GET that rendered the form and the POST that submitted it reached JupyterHub from two different “client\n IPs”.<\/li>\n<\/ul>\n
        Fix:<\/strong> externalTrafficPolicy: Local<\/code>. Traffic now enters only on the node running Traefik, the real client IP is\npreserved, and logins went 10\/10. The verification script now logs in three times in a row. A single success proves\nnothing for an intermittent bug.<\/p>\n
        Apps that don’t trust proxies<\/h3>\n
        Putting apps behind a reverse proxy surfaces each app’s opinion about X-Forwarded-*<\/code> and Host<\/code> (#96, #99):<\/p>\n
          \n
        • Trino<\/strong> returned 406<\/code> until http-server.process-forwarded=true<\/code>.<\/li>\n
        • MLflow 3<\/strong> has a host-header allow-list (serverAllowedHosts<\/code>) and returned 403<\/code> for the new name.<\/li>\n
        • NiFi 2<\/strong> enforces SNI against its own certificate. That certificate is generated at first start from the proxy\n hosts it knows then<\/em>. Adding a name later means regenerating the keystore. The documented “disable SNI check”\n switch has no effect in 2.12.<\/li>\n<\/ul>\n
          None of these are bugs. They are security features doing their job, and they all fail closed. Expect one per app.<\/p>\n
          NetworkPolicies cut both ways<\/h3>\n
          JupyterHub’s chart ships a NetworkPolicy that blocks notebook egress to private IPs<\/strong>. In a cluster, that’s every\nService (#71). From a notebook, MLflow, Trino and S3 all timed out.<\/p>\n
          The tempting fix (privateIPs: true<\/code>) would also have opened the entire home LAN to every notebook. Instead the policy\nallows egress to the data-platform namespaces only, so each new namespace has to be added deliberately. Default-deny\nwith explicit allows is the model worth following everywhere.<\/p>\n
          Checklist<\/h3>\n
            \n
          1. Choose the LB range and remove it from DHCP<\/strong>: pool and<\/em> reservations.<\/li>\n
          2. Whitelist every split-horizon domain<\/strong> in the router’s rebind protection, and in its restore script.<\/li>\n
          3. Use DoH (or a resolver you control) for ACME DNS-01 checks<\/strong> if anything on the LAN intercepts port 53.<\/li>\n
          4. externalTrafficPolicy: Local<\/code><\/strong> on the ingress Service if any app ties sessions to client IPs, or if you want\n real IPs in your logs.<\/li>\n
          5. Force IPv4 for SSH and Git<\/strong> in pods if the cluster has no IPv6.<\/li>\n
          6. Per app:<\/strong> forwarded headers, allowed hosts, SNI.<\/li>\n
          7. Wildcard DNS + wildcard certificate + one Gateway<\/strong>: adding an app should be one HTTPRoute and nothing else.<\/li>\n<\/ol>\n
            Previous: Part 1 (one spinning disk \u00b7 Next: Part 3) Adopting a live cluster into\nGitOps.<\/em><\/p>\n
            \n\n\n\n
            Get the next one in your inbox<\/h2>\n\n\n\n
            Enjoyed this? I write about local AI, self-hosting, and cloud cost ninjutsu. Join the Ninja Dispatch for occasional, high-signal notes. No fluff.<\/p>\n\n\n\n\n\t\t\t