I’ll just provide my own example: my homelab consists of 6 Kubernetes nodes placed across the country. Some differ by ISP, some are placed in different cities, one is hosted on a cloud provider. Basically it’s a very cheap variant of geo-replicating my workloads.

Two of these nodes are visible from the Internet and have a static IP address; one node also has an IPv6 address. Each node hosts an authoritative DNS server (CoreDNS) for my personal domain pootis.network; and the .network TLD has glue records which point to IPs of these two nodes. This is a classic “self-hosted DNS” scenario.

Here’s an excerpt from my zonefile so you can understand the setup better:

$ORIGIN pootis.network.
$TTL 300

@       SOA     ns1.pootis.network. admin.pootis.network. (
  2026082001
  1200
  300
  1209600
  300
)

; Nameservers and glue records
@       NS      ns1.pootis.network.
@       NS      ns2.pootis.network.
ns1     A       178.44.116.85
ns2     A       91.219.150.30
ns2     AAAA    2a06:dd00:1:4::4189

This 5-record block (NS/A/AAAA) is mirrored into the .network zone by my domain registrar (plus DS for DNSSEC but that’s another thing).

As such, my DNS becomes fully independent - and, in theory, if one of my externally-facing nodes breaks, let’s say ns1, then DNS resolvers all over the world (forwarders, recursive, and such) will fall back to ns2, and everything will keep working. Kubernetes will also reorganize the pod placement so all my workloads are available again after a slight downtime.

That would have been great, if it worked as described, but apparently, after one nameserver in my zone fails, then the resolvers… just give up? Let’s say ns1 failed but ns2 is working. The parent zone still points to both nameservers. My external resource records (websites and other stuff) at this point would have already been auto-reconfigured by a custom k8s controller to point to the IP addresses of the node that hosts ns2. Simplifying: the entire world basically sees this after ns1 fails and after TTL caches expire:

; all of this has very low TTL, 5 minutes or so

@       NS      ns1.pootis.network. ; from .network 
@       NS      ns2.pootis.network. ; from .network

ns1     A       178.44.116.85 ; broken. Either from .network glue or from my auth DNS
ns2     A       91.219.150.30 ; either from .network glue or from my auth DNS
ns2     AAAA    2a06:dd00:1:4::4189 ; same

; my-website     A       178.44.116.85 ; does not appear because ns1 is broken- my LB already removed it from the set
my-website     A       91.219.150.30 ; fronted by a pair of CNAMEs due to loadbalancing but still
my-website     AAAA    2a06:dd00:1:4::4189 ; same

But even if I query 1.1.1.1 directly for my-website’s record, it just doesn’t work most of the time because the resolver pins itself to ns1 which is currently failing, or it selects ns1 and does not even care to try ns2.

To be precise: some resolver implementations DO fall back to ns2 as expected, but most of them just pin themselves to ns1 and then outright refuse to resolve the records in my zone.

And there’s actually no reasonable way out, as far as I can see:

  • moving my DNS infra somewhere else (CloudFlare, for example) is unacceptable since I would like for my homelab to be as independent as practically possible;
  • anycasting, or running a fully-fledged BGP AS is also impossible because that costs a lot of money and I’d like for my homelab to fit into a $10/month budget with room to spare;
  • “live-patching” the NS and glue records in the parent zone (.network), to keep up with the set of my working nodes, is possible, but very unwieldy and somewhat hard to accomplish.

There’s a lot of custom machinery that keeps my workloads running and accessible after a node failure, but all of this becomes completely moot when authoritative DNS is the bottleneck.

Has anyone been running a similar stack and encountered this problem? I’m aware that the answer is usually “host your DNS at CloudFlare” or “use the registrar’s DNS infra” but still…