Comment - FBXL Lotide

Comment on How do people handle authoritative DNS redundancy for their self-hosted workloads?

mschae@discuss.mschae23.de ⁨5⁩ ⁨hours⁩ ago

But even if I query 1.1.1.1 directly for my-website’s record, it just doesn’t work most of the time because the resolver pins itself to ns1 which is currently failing, or it selects ns1 and does not even care to try ns2.

That sounds bad. They really shouldn’t do that.

moving my DNS infra somewhere else (Cloudflare, for example)

And it would also mean this isn’t a solution either. If the problem is on the resolvers’ side, moving to a different DNS hoster would not change anything. Hm. And the other two potential solutions you’ve listed would be extremely overkill for this use case…

Personally, I only host one authoritative namserver and do redundancy by using some free secondary DNS services (although a friend has also recently set up an authoritative nameserver, so maybe we’ll host secondary DNS for each other in the future). I haven’t encountered this issue before, but my server also hasn’t been down a lot.

There’s a lot of custom machinery that keeps my workloads running and accessible after a node failure, but all of this becomes completely moot when authoritative DNS is the bottleneck.

If it’s only necessary for your own stuff, maybe you should set up a local DNS resolver that works correctly which your services can use?

original

Sort:hotnew top

dave@lemmy.pootis.network ⁨5⁩ ⁨hours⁩ ago
I actually already have a local DNS resolver… well, sort of: most of my cluster’s internal services, such as Vaultwarden and other stuff, are available only over an in-cluster IKEv2 VPN.

What I mean is, I have to connect to the VPN first, which pushes a Configuration Payload to my IKEv2 client, and that payload contains a DNS server IP, and that DNS server (exposed by a Kubernetes Service) serves a custom “internal” zone (.int.pootis.network), and proxies everything else either to k8s coredns, or to upstream DNS forwarders.

Okay, that explanation may have been somewhat complicated (it sounded simpler in my head) but the end result is that I can resolve and reach stuff like lemmy.int.pootis.network (my Lemmy frontend) only while connected to the in-cluster VPN. It is completely hidden from external users (no CT log record, no external DNS record, and only proxyable by internal Traefik instance).

The issue only occurs with my externally-facing DNS, which is needed to, for example, to reach my Lemmy backend instance (lemmy.pootis.network) and the pictrs deployment. But yeah, internal DNS works perfectly for me and it easily survives a node failure.

original
- dave@lemmy.pootis.network ⁨4⁩ ⁨hours⁩ ago
  Basically my internal DNS zonefile is configured like this (it should be easier to read than my explanation):
  
  coredns_config: zoneFiles: - filename: int.pootis.network.zone domain: int.pootis.network contents: | $ORIGIN int.pootis.network. $TTL 300 @ SOA ns.int.pootis.network. admin.pootis.network. ( 2026082001 1200 300 1209600 300 ) @ NS ns.int.pootis.network. ns AAAA {{ k8s_dns_ipv6 }} ns A {{ k8s_dns_ipv4 }} vault CNAME internaltraefik.internaltraefik.svc.{{ k8s_cluster_domain }}. cinny CNAME internaltraefik.internaltraefik.svc.{{ k8s_cluster_domain }}. grafana CNAME internaltraefik.internaltraefik.svc.{{ k8s_cluster_domain }}. stalwart CNAME internaltraefik.internaltraefik.svc.{{ k8s_cluster_domain }}. lemmy CNAME internaltraefik.internaltraefik.svc.{{ k8s_cluster_domain }}. mail CNAME stalwart.stalwart.svc.{{ k8s_cluster_domain }}. webmail CNAME internaltraefik.internaltraefik.svc.{{ k8s_cluster_domain }}. ; ACME challenges ; int.pootis.network + *.int.pootis.network _acme-challenge CNAME 1ca12d2c-8034-4a09-a940-dc59bd4f38d8.acme-dns.pootis.network. ; mail.int.pootis.network _acme-challenge.mail CNAME d175b66d-909f-46ba-8572-636ebc235d4c.acme-dns.pootis.network.
  
  original
  - mschae@discuss.mschae23.de ⁨4⁩ ⁨hours⁩ ago
    Interesting, that’s much more sophisticated than my setup (though to be fair, I host my services on a rented VPS instead of home)!
    
    The issue only occurs with my externally-facing DNS, which is needed to, for example, to reach my Lemmy backend instance (lemmy.pootis.network) and the pictrs deployment. But yeah, internal DNS works perfectly for me and it easily survives a node failure.
    
    Could still set up a recursive resolver for external DNS, but I guess that wouldn’t help other people who use a bad resolver. Tricky situation.
    
    original
    dave@lemmy.pootis.network ⁨4⁩ ⁨hours⁩ ago
    Yeah, the issue is that people’s resolvers are wildly different and I guess some of them do not cater to weird and wacky self-hosting setups (most people just use anycast highly-available DNS, usually provided by a registrar or by some other company).
    
    I had a hunch that resolvers break because one of my nodes does not have a public IPv6 address: meaning,
    
    node2: externally-facing node, I host it from my home, IP 178.44.116.85, acts as NS ns1. It has IPv6 connectivity but no external IPv6;
    
    node4: a cloud VPS that I rent. Externally-facing, with IPv4 91.219.150.30 and IPv6 2a06:dd00:1:4::4189, acts as NS ns2.
    
    If node4 decides to break, then ns2 becomes unavailable - but since resolvers can see that ns1 does not have an IPv4 address, and since everyone knows that IPv6 is better because 6 is greater than 4, then these resolvers would always prefer ns2, even though it’s clearly down.
    
    This could be solved by asking my ISP to set up a static IPv6 prefix for me (they already provide static IPv4), except my ISP does not do that, unfortunately. I could also rent another VPS (so I would have two VPS’es, giving me some resilience from downtimes) but that means extra $$$… anyway, I guess I should probably test the IPv6-preference hypothesis first and then act on the results.
    
    original