DNS Works on My Laptop: CoreDNS Failures in Kubernetes

It was a Wednesday, mid-morning, when support tickets started describing the API as "flaky," and my laptop refused to reproduce any of it.

From my machine: every endpoint fast, every lookup instant. From inside the cluster: every few dozen requests, one would hang for exactly five seconds, then succeed. Five seconds. If you've chased in-cluster DNS before, that number is already telling you the answer. I hadn't, yet. I spent the first hour blaming the ingress.

"DNS works on my laptop" is the most useless sentence in Kubernetes debugging. Your laptop isn't using CoreDNS, isn't using cluster search domains, and isn't going anywhere near conntrack. The pod is doing all three.

Quick answer: In-cluster DNS failures come from three places: ndots:5 query amplification making CoreDNS do five times the work, the UDP conntrack race dropping packets and causing five-second stalls, or CoreDNS itself being overloaded or unhealthy. Confirm the symptom from inside a pod with dig, check CoreDNS pods in kube-system for restarts and throttling, and read the pod's /etc/resolv.conf to see how many search-domain lookups each name is generating. The durable fix for the five-second stalls is NodeLocal DNSCache.

Reproduce it from inside the cluster

First rule: debug DNS from a pod, not from your workstation. Spin up a throwaway pod and make the resolver show its work:

$ kubectl run dnsdebug --rm -it --image=busybox:1.36 --restart=Never -- sh
/ # cat /etc/resolv.conf
search prod.svc.cluster.local svc.cluster.local cluster.local
nameserver 10.96.0.10
options ndots:5
/ # nslookup api.example.com
;; connection timed out; no servers could be reached

That resolv.conf is the whole game. nameserver 10.96.0.10 is the kube-dns Service IP -- a virtual IP that kube-proxy DNATs to a CoreDNS pod. And ndots:5 with three search domains is about to multiply every lookup your app makes.

If a direct query to CoreDNS works (nslookup kubernetes.default 10.96.0.10) but normal lookups stall, the service path is fine and you should suspect amplification or packet drops. If even direct queries fail, check whether the CoreDNS pods are alive before anything else.

ndots:5, the multiplier nobody reads

Here's what ndots:5 actually does: any name containing fewer than five dots is treated as possibly-internal, so the resolver tries it against every search domain *first*, and only tries the bare name after all of those fail. With the default three search domains, resolving api.example.com -- two dots -- produces:

api.example.com.prod.svc.cluster.local.   NXDOMAIN
api.example.com.svc.cluster.local.        NXDOMAIN
api.example.com.cluster.local.            NXDOMAIN
api.example.com.                          NOERROR

Four queries where your laptop made one, three of them guaranteed garbage. Now multiply that by every HTTP client in every pod, every request, and you see how a cluster ends up doing 50,000 DNS QPS to resolve a few thousand actual names. CoreDNS melts, latency climbs, and everyone's laptop keeps working fine.

Two fixes, use both. In application config, terminate external hostnames with a trailing dot (api.example.com.) so the resolver skips the search list entirely. And for pods that mostly talk to the outside world, tune the resolver directly:

spec:
  dnsConfig:
    options:
      - name: ndots
        value: "2"

Don't crank ndots down cluster-wide without thinking, though. Internal short names like redis genuinely rely on search-domain expansion. That one weird trick -- setting ndots:0 for a pod that only calls external APIs -- trades away exactly the feature it never uses.

The five-second stall: conntrack's race

The Wednesday tickets, though, were the other classic. UDP DNS from a pod goes through kube-proxy's DNAT, and the kernel's conntrack table has a race when two UDP flows to the same destination start simultaneously. One packet loses the race and is silently dropped. No ICMP error, no log, just silence. The resolver waits its default timeout -- five seconds -- and retries, usually over TCP or to the second query in the batch, and succeeds.

Hence "flaky." Hence exactly five seconds. Hence un-reproducible from your laptop, which doesn't DNAT anything.

You can confirm it by watching for retransmits on a busy node, but honestly the pattern is diagnostic enough: intermittent stalls, always the same duration, under load, gone at 3am. The fixes, in order of permanence:

NodeLocal DNSCache -- a DaemonSet that runs a caching resolver on each node, reachable over a link-local address with no DNAT and no conntrack race. This is the real fix and what every large cluster ends up running.
single-request-reopen or use-vc in the pod's dnsConfig options -- works around the race per-pod by changing how the resolver sends queries. A bridge, not a destination.
Reduce query volume with the ndots fixes above -- fewer packets, fewer races.

When CoreDNS actually is the problem

Sometimes it isn't the path. Check the pods themselves:

$ kubectl get pods -n kube-system -l k8s-app=kube-dns
NAME                       READY   STATUS    RESTARTS        AGE
coredns-7db6d8ff4d-9q2wz   1/1     Running   14 (3d ago)     21d
coredns-7db6d8ff4d-k4xvt   1/1     Running   0               21d

Fourteen restarts on one replica is a story -- read kubectl logs -n kube-system coredns-7db6d8ff4d-9q2wz --previous and check whether it's OOMKilled. Also check CPU: CoreDNS replicas that are CPU-throttled answer slowly, and "slow DNS" is indistinguishable from "dropped DNS" from the application's side.

Scaling is the unglamorous fix that works. Two CoreDNS replicas is a default, not a plan. The upstream cluster-proportional-autoscaler grows CoreDNS with cluster size, and at real scale NodeLocal takes most of the load off CoreDNS entirely. If you're running Prometheus, coredns_dns_requests_total and coredns_panics_total will tell you whether you're anywhere near the ceiling before your users do.

Wednesday's resolution: ndots amplification from one chatty service had tripled cluster QPS, the extra UDP volume made the conntrack race fire constantly, and the throttled CoreDNS pair turned drops into stalls. We deployed NodeLocal DNSCache, put a trailing dot on the worst offender's upstream hostname, and scaled CoreDNS to four. The five-second stalls never came back.

Frequently asked questions

Why does DNS work on my laptop but fail inside a Kubernetes pod?

Pods use the cluster DNS service, usually CoreDNS behind the kube-dns Service IP, not your host resolver. Pod resolv.conf also carries search domains and ndots:5, so a single lookup inside a pod expands into multiple queries that your laptop never makes. A failure can be CoreDNS itself, the network path to it, or query amplification overwhelming it, none of which affects your laptop at all.

What is ndots:5 and why does it slow down DNS in Kubernetes?

ndots:5 means any name with fewer than five dots is treated as potentially internal, so the resolver tries it against every search domain before trying the name as-is. A name like api.example.com has two dots, so it gets tried against three or four cluster search domains first, generating several wasted NXDOMAIN queries per lookup. The fix is a trailing dot on fully qualified names or a lower ndots value in the pod's dnsConfig.

What causes intermittent five-second DNS timeouts in Kubernetes?

The classic cause is a conntrack race: UDP DNS packets going through kube-proxy DNAT can hit a race in the kernel's connection tracking table, silently dropping packets. The resolver waits five seconds and retries, which looks like random slowness. The durable fixes are NodeLocal DNSCache, which talks to the node directly over a link-local address, or forcing single-request behavior in dnsConfig options.

How do I check if CoreDNS itself is the problem?

Run kubectl get pods -n kube-system -l k8s-app=kube-dns and check for restarts, OOMKills, and CPU throttling, then read the logs with kubectl logs on the CoreDNS pods. From inside a debug pod, query the CoreDNS service IP directly with dig and compare latency against the same query sent to an upstream resolver. If direct queries are fast and service-path queries are slow, the problem is the network path, not CoreDNS.

Should I use NodeLocal DNSCache?

If your cluster has high DNS query volume, heavy use of external hostnames, or intermittent UDP timeout complaints, yes. NodeLocal runs a DNS cache as a DaemonSet on every node, so pods resolve locally instead of traversing DNAT to a shared CoreDNS service, which removes the conntrack race and absorbs query storms. It is the standard fix at scale and is upstream Kubernetes, not a vendor add-on.

The takeaway

In-cluster DNS is a different system than the one on your laptop, with its own failure modes: search-domain amplification, conntrack races, and two little CoreDNS pods doing more work than anyone realized. Debug from inside a pod, count the queries, and check the resolvers before the ingress.

And when the stalls are exactly five seconds, stop packet captures and go deploy NodeLocal DNSCache. The kernel already told you who did it.