Publishing 13 web applications from a Kubernetes cluster under proper HTTPS names sounds like a solved problem, and in a cloud VPC it mostly is. Managed load balancers, managed DNS and predictable resolvers hide most of the work.

This cluster ran on a private network with none of those services, as part of the lakehouse exercise described in “When Storage Is the Bottleneck”. Every layer a cloud provider would normally abstract away had to be built and understood directly. That made it a useful way to see where the common assumptions about Kubernetes networking stop holding.

The architecture

client
  -> internal DNS: *.apps.example.com resolves to the reverse proxy
  -> Nginx Proxy Manager (holds the *.apps.example.com certificate)
  -> Traefik on a MetalLB address (SNI = requested host)
  -> shared Gateway -> one HTTPRoute per application -> ClusterIP Service

Each layer was a deliberate choice:

• Gateway API instead of Ingress. ingress-nginx was retired upstream in March 2026, and Traefik has become a common default. Gateway API also separates responsibilities: the platform owns one shared Gateway and its wildcard certificate, while each namespace attaches its own HTTPRoute. A namespace opts in through a label (gateway.example.com/routes=true), so one tenant cannot take over another tenant’s hostname.
• MetalLB in L2 mode gives Traefik a routable address on the network, with one node answering ARP for it. This fills the role of a cloud load balancer.
• One wildcard certificate from Let’s Encrypt, using DNS-01 through cert-manager and a Cloudflare API token. No application needs its own certificate, and nothing has to be reachable from the internet to obtain one.
• A single reverse proxy in front. Internal traffic and Cloudflare Tunnel traffic share one entry point, so there is only one place to secure. With a wildcard at both the DNS and proxy layers, a new application needs only an HTTPRoute, with no DNS record and no proxy change.

The design worked. The rest of this article covers the places where the network behaved differently from what the design assumed.

Trap 1: DNS interception broke certificate issuance

The certificate would not issue. cert-manager reported “DNS record not yet propagated”, even though the TXT record was plainly visible in the Cloudflare dashboard. Querying Cloudflare’s authoritative nameservers directly from inside the network returned NXDOMAIN.

The response flags gave it away. The answer carried rd ra and no aa, so it had not come from an authoritative server. The network was intercepting all DNS traffic on port 53, regardless of destination, which is common with content filtering and some gateways. The internal resolver answered instead, and it knew nothing about _acme-challenge.

The fix was to have cert-manager check propagation over DNS-over-HTTPS, which this kind of interception cannot touch:

--dns01-recursive-nameservers-only
--dns01-recursive-nameservers=https://cloudflare-dns.com/dns-query,https://dns.google/dns-query

The certificate issued within two minutes and has renewed automatically since.

Trap 2: Rebind protection discarded valid answers

Names resolved correctly on clients that queried the internal resolver directly, and failed on clients that resolved through the router.

The router runs dnsmasq with rebind_protection. When a public domain returns a private (RFC 1918) address, dnsmasq treats it as a DNS rebinding attack and strips the answer. The client receives NOERROR with zero records and gives up, which makes the failure easy to misread.

The fix is one line per internal domain, rebind_domain='<your-domain>', added to both the live configuration and the router’s configuration restore script. The second step matters: without it, a firmware upgrade silently removes the setting, and this had already happened once.

The same issue applies to Cloudflare Tunnel. Public DNS points at Cloudflare, so everything works from outside, while the same names fail internally until the exception is in place. “Works outside, broken inside” is the signature of split-horizon DNS meeting rebind protection.

Trap 3: A forwarder pointing at itself

The apps.example.com zone on the internal resolver was a forwarder zone, and its forwarder was the resolver itself. Any name without a local record, including the apex and www, returned SERVFAIL internally. The fix was real upstream forwarders plus a single wildcard record pointing at the reverse proxy.

An apex record pointing at the proxy was tested and rolled back, because the proxy had no host for the apex and internal clients lost access to the public website. Wildcards do not cover the apex, and that is usually the behaviour you want.

Trap 4: IPv6 that was not there

Argo CD could not fetch a 180 KB Git repository, failing with git fetch ... timeout after 1m40s. The pods had IPv4 only, but the Git host resolved to an IPv6 address first. OpenSSH waited on a connection that could never succeed, and Argo CD’s 90-second exec timeout expired before it fell back.

A mounted SSH client configuration with AddressFamily inet resolved it. This is a sensible default for any IPv4-only pod network.

An intermittent login failure with a network root cause

About half of all JupyterHub logins through the Gateway failed with “Login form invalid or expired”, with the same browser and the same password. The failure looked like an application bug. It was caused by the traffic path:

• JupyterHub binds its anonymous XSRF token to the client IP and user agent.
• Traefik’s Service used externalTrafficPolicy: Cluster. MetalLB delivers traffic to one node, and kube-proxy may forward it to a Traefik pod on another node, rewriting the source address along the way.
• As a result, the GET that rendered the form and the POST that submitted it could reach JupyterHub from two different client IPs.

Setting externalTrafficPolicy: Local preserved the real client IP, and logins succeeded 10 out of 10 times. The verification script now logs in three times in a row, because a single success proves nothing for an intermittent bug.

Applications have their own proxy rules

Placing applications behind a reverse proxy exposes each one’s rules about X-Forwarded-* headers and Host:

• Trino returned 406 until http-server.process-forwarded=true was set.
• MLflow 3 enforces a host allow-list (serverAllowedHosts) and returned 403 for the new name.
• NiFi 2 validates SNI against its own certificate, which is generated at first start from the proxy hosts it knows at that moment. Adding a name later means regenerating the keystore, and the documented switch to disable the SNI check had no effect in version 2.12.

None of these are bugs. They are security controls doing their job, and they all fail closed. Plan for one per application.

NetworkPolicies cut both ways

JupyterHub’s chart ships a NetworkPolicy that blocks notebook egress to private IP ranges. Inside a cluster, that covers every Service, so MLflow, Trino and S3 all timed out from notebooks.

The quick fix, privateIPs: true, would also have opened the entire internal network to every notebook. Instead, the policy allows egress to the data platform namespaces only, and each new namespace has to be added deliberately. Default-deny with explicit allows is the right model everywhere.

Recommendations

1. Reserve the load balancer range and exclude it from DHCP, covering both the pool and any reservations.
2. Add every split-horizon domain to the router’s rebind exceptions, including its restore configuration.
3. Use DNS-over-HTTPS or a resolver you control for ACME DNS-01 checks if anything on the network intercepts port 53.
4. Set externalTrafficPolicy: Local on the ingress Service if any application ties sessions to client IPs, or if you want real addresses in your logs.
5. Force IPv4 for SSH and Git in pods when the cluster has no IPv6.
6. Check three things per application: forwarded headers, allowed hosts and SNI.
7. Combine wildcard DNS, a wildcard certificate and one Gateway, so that adding an application takes one HTTPRoute and nothing else.

The common thread is that each failure sat between layers. Each component was working as designed, and the fault only appeared in how they interacted. That is why flags, source addresses and packet paths told us more here than application logs did.

---

Get the next one in your inbox

Enjoyed this? I write about local AI, self-hosting, and cloud cost ninjutsu. Join the Ninja Dispatch for occasional, high-signal notes. No fluff.

No spam. Unsubscribe anytime.