26 Jul 2026 · 32 min read

It is probably not DNS

"It's always DNS" is a punchline that became a diagnostic strategy. Six unrelated faults — ephemeral ports, conntrack, MTU, egress policy, CPU throttling, kube-proxy rule scaling — produce a symptom indistinguishable from a DNS failure, and each has one observation that rules it out.

“It’s always DNS” is a good joke. The problem is that it stopped being a joke and became a diagnostic strategy.

The mechanism is ordinary confirmation bias with a very short feedback loop. You read one excellent write-up about a DNS bug — the conntrack race, the ndots amplification, a CoreDNS pod that was quietly OOMKilled — and the next time something is intermittently slow you have a hypothesis pre-loaded before you have a single observation. It is available, it is cheap to state, and it is socially safe: nobody has ever been mocked in a postmortem for suspecting DNS first.

What makes it expensive is that a stack of unrelated failures all present identically:

intermittent timeouts, not consistent ones
works, then doesn’t, then works again
fine from one pod, broken from another
fine for small requests, broken for large ones
fine at 09:00, broken at 09:15 under load

Every one of those is compatible with a DNS fault. Every one of them is also compatible with at least four other faults that have nothing to do with name resolution. And DNS has a particular property that makes it worse: it is the most visible casualty of several lower-level problems. When conntrack drops packets, the thing that visibly breaks is a UDP DNS lookup, because DNS is the highest-frequency, most latency-sensitive UDP traffic in the cluster. So you do find a DNS symptom. You find it because it is downstream of something else.

This post is not about DNS. It is about the elimination method — how to spend twenty minutes narrowing the search instead of two hours confirming a hypothesis you started with.

Part 1: the impostors

For each of these I care about three things: the mechanism, what it looks like from inside a pod, and — the part that actually matters — the single observation that distinguishes it from a genuine DNS fault. If you take one thing from this post, take that column.

1.1 Ephemeral port and connection-pool exhaustion

Mechanism. Every outbound connection needs a source port. Linux allocates from net.ipv4.ip_local_port_range, by default 32768 60999 — about 28,000 ports. When a connection closes, the initiating side holds the tuple in TIME_WAIT for 2×MSL (60s on Linux) so that late duplicates cannot be mistaken for a new connection. Under SNAT — which is what happens to pod traffic leaving the node in most CNI configurations — the constraint tightens: the tuple that must be unique is (source IP, source port, destination IP, destination port), and if a large number of pods behind one node address are all talking to the same destination, they are competing for the same port space.

Connection pool exhaustion is the application-layer twin: an HTTP client with maxConnsPerHost or a database pool at its limit will block new requests until a connection frees up. Same symptom, different layer.

From inside the pod. Requests hang and then time out. Latency is fine — excellent, even — for the requests that succeed, and infinite for the ones that don’t. Failure rate scales with throughput rather than being constant. Below some request rate everything is perfect; above it, a percentage of requests fail. Restarting the pod fixes it for a while.

The discriminating observation. EADDRNOTAVAIL — “cannot assign requested address” — in the application log or under strace. That error comes from bind()/connect() inside your own netns; it never touches the network, and it is not a thing a DNS failure can produce. Failing that, count sockets:

# in the pod's netns, or on the node for the SNAT case
ss -s
ss -tan state time-wait | wc -l
cat /proc/sys/net/ipv4/ip_local_port_range

A TIME_WAIT count in the tens of thousands against a 28,000-port range is not a hint, it is the answer. The second discriminator: the failure is destination-specific. Port exhaustion against one busy backend does not stop you resolving or reaching a different one. A broken resolver stops everything.

1.2 Conntrack exhaustion, and the DNAT insertion race

This one deserves the most space, because it is the actual reason everybody blames DNS.

Mechanism, part one — the table. nf_conntrack keeps a table of tracked flows. It is finite. kube-proxy sizes it on startup: --conntrack-max-per-core defaults to 32768 and --conntrack-min to 131072, so the effective nf_conntrack_max is max(32768 × cores, 131072). Every Service connection, every NodePort flow and every UDP exchange consumes an entry. UDP entries are the nasty ones: a UDP “flow” has no close, so the entry lingers for nf_conntrack_udp_timeout — 30 seconds by default — after a single DNS query. A pod doing a few hundred lookups a second holds thousands of dead UDP entries at any moment. When the table fills, the kernel drops packets and logs nf_conntrack: table full, dropping packet.

Mechanism, part two — the race. This is the famous one. A Service ClusterIP is implemented by DNAT: the packet leaves the pod addressed to 10.96.0.10:53 and iptables rewrites the destination to a CoreDNS pod IP. DNAT and conntrack insertion are not atomic. glibc’s resolver sends the A and AAAA queries in parallel, from the same socket, which means two UDP packets with identical source tuples enter the netfilter path within microseconds of each other. Neither finds a confirmed conntrack entry, both create one, and at insertion time one of them loses and is dropped. The application sees no reply, and the resolver’s default timeout is 5 seconds — hence the canonical symptom: a DNS lookup that takes exactly five seconds, occasionally, for no reason.

That is kubernetes#56903, and I come back to it in detail in Part 4, including what is still true.

From inside the pod. Timeouts at suspiciously round intervals (5s, 10s). Affects everything on the node, not one workload. Correlates with load. Getting worse over hours, then abruptly fine after a node reboot.

The discriminating observation. Two counters, on the node, not in the pod:

# how full is the table
sysctl net.netfilter.nf_conntrack_max
cat /proc/sys/net/netfilter/nf_conntrack_count

# the race, specifically
conntrack -S | tr ' ' '\n' | grep -E 'insert_failed|drop|early_drop'

insert_failed climbing is the fingerprint of the insertion race. nf_conntrack_count approaching nf_conntrack_max, plus nf_conntrack: table full in dmesg, is exhaustion. These are different problems with different fixes and the same symptom, so read both.

Crucially, neither counter says anything about DNS. If insert_failed is climbing, your DNS symptom is a consequence; fixing CoreDNS will do nothing. And the inverse holds: a genuine DNS fault will not move these counters at all.

1.3 MTU mismatch between overlay and underlay

Mechanism. An encapsulating CNI wraps every pod packet in an outer header — VXLAN, Geneve, IPsec, WireGuard. That overhead has to come out of the payload, so the pod-facing MTU must be lower than the underlay MTU. Cilium detects the underlying device MTU and subtracts the tunnel overhead automatically, and gets it right in the ordinary case. It gets it wrong when the underlay is not what it looks like: a cloud VPC with a lower MTU than the NIC advertises, a VPN or transit hop in the middle, jumbo frames on one node and not another, or a node added later with a different NIC configuration.

Then there is a second failure that turns a small problem into a total one. The correct behaviour when a too-large packet with DF set hits a link is an ICMP “fragmentation needed” (type 3, code 4) back to the sender — Path MTU Discovery. If something in the path drops ICMP — a security group, a firewall, a load balancer, a well-meaning hardening rule — PMTUD fails silently and the sender keeps retransmitting a packet that can never arrive. This is the PMTUD black hole, and it is the single most misdiagnosed condition in cluster networking.

From inside the pod. This is the distinctive one. The TCP handshake succeeds — SYN, SYN-ACK and ACK are tiny. The connection is established. Then the first full-size data segment vanishes and the connection hangs until something times out. So:

curl to a small endpoint (/healthz returning ok) works perfectly
curl to an endpoint returning 40 KB of JSON hangs after headers
git clone of a small repo works, of a large one stalls at “Receiving objects”
kubectl exec works; kubectl logs on a chatty pod hangs
docker pull / image pulls hang partway

And DNS works fine, because a DNS response is usually under 512 bytes. Which is exactly why people rule out DNS, correctly, and then still fail to find it — the symptom is size-dependent, and nothing about “intermittent hang” suggests measuring packet size.

The discriminating observation. Binary-search the payload size with a don’t-fragment ping:

# 1472 = 1500 - 20 (IP) - 8 (ICMP). If 1472 fails and 1400 passes, you have your answer.
ping -M do -s 1472 -c 2 <peer-pod-ip>
ping -M do -s 1400 -c 2 <peer-pod-ip>

The signature of an MTU problem is a clean threshold: every size at or below N passes, every size above N fails, reproducibly. No DNS fault in the world produces a size-dependent cliff. If you can find the cliff, you are done — you now know the real MTU and you can compare it against what the CNI configured.

1.4 NetworkPolicy or a security group blocking egress

Mechanism. A default-deny egress NetworkPolicy is one of the first things a serious platform team applies, and it is the correct thing to do. It is also the single most common way to accidentally break DNS, because DNS egress is a dependency that no application manifest declares. You allow egress to the payments namespace, you deploy, and the pod cannot resolve anything — including the payments service you just allowed — because UDP/53 to kube-system was never in the policy. Cloud security groups and NACLs produce the same thing one layer down, as do node-level firewall rules.

From inside the pod. Everything by name fails, fast or slow depending on whether the packet is dropped (timeout) or rejected (immediate error). Traffic to hard-coded IPs may work perfectly, which is confusing until you realise it is the diagnosis.

The discriminating observation. Two:

The blast radius is per-pod, not per-node. Another pod on the same node, in a different namespace or with different labels, is fine. Node-level faults (conntrack, MTU, kube-proxy) do not respect namespaces. Policy faults do exactly that.
Drop versus reject timing. A NetworkPolicy drop gives you a hang and a timeout. A security group gives you a hang. A rejecting firewall gives you an instant connection refused. Instant failure is almost never DNS; DNS failures cost you the resolver timeout.

The confirming check is to enumerate what actually applies to the pod, rather than reading the policy you think you wrote:

# every policy whose selector could match this pod's labels
kubectl get networkpolicy -n <ns> -o yaml

# does the pod even have a route out, ignoring names entirely
kubectl exec -n <ns> <pod> -- nc -zv -w 3 <a-known-good-ip> 443

1.5 CPU throttling on the calling pod

Mechanism. A CPU limit is implemented as a CFS quota: the cgroup gets quota microseconds of CPU per 100 ms period, and when it exhausts them every thread is descheduled until the period rolls over. A pod with limits.cpu: 500m gets 50 ms per 100 ms, and if it burns through that in the first 30 ms it does nothing at all for the remaining 70 ms. Multi-threaded runtimes make this dramatically worse: a JVM or a Go program that thinks it has 16 cores will consume a 500m quota in a few milliseconds of wall time.

Now consider what a stalled thread looks like. The request was sent. The reply arrived and is sitting in a socket buffer. Nobody reads it for 70 ms. If the application’s client timeout is tight, it declares a timeout — for a request that succeeded. If the stall happens between sendto() and recvfrom() on a DNS socket, the resolver declares a DNS timeout. The network was never involved.

From inside the pod. Timeouts that correlate with the pod’s own busy periods. p99 latency far worse than p50 with no corresponding server-side latency. Absolutely nothing wrong at the other end — the server logs show the request handled in 3 ms.

The discriminating observation. Read the cgroup’s own throttling counters, which are authoritative and cost nothing:

# cgroup v2, from inside the pod
cat /sys/fs/cgroup/cpu.stat
# nr_periods, nr_throttled, throttled_usec

# cgroup v1
cat /sys/fs/cgroup/cpu/cpu.stat

Or, from the metrics side, container_cpu_cfs_throttled_periods_total over container_cpu_cfs_periods_total. If nr_throttled is a meaningful fraction of nr_periods during the incident window, you have found a self-inflicted latency problem and there is nothing to fix in the network. The clinching detail: compare the client’s measured latency to the server’s. If the server says 3 ms and the client says 900 ms, the time was spent in a queue somewhere, and the runqueue of a throttled cgroup is the cheapest place to look.

1.6 kube-proxy rule-set size at high Service counts

Mechanism. kube-proxy’s iptables mode writes a rule chain per Service, and the packet path evaluates them linearly. The ruleset grows with (Services + endpoints), so per-packet latency in the worst case grows O(n) in the number of Services. Worse than the data-plane cost is the control-plane cost: iptables has no incremental update, so each change means reading, rewriting and reloading a ruleset that can reach hundreds of thousands of rules. In large clusters that turns into multi-second iptables-restore cycles and Service updates that take minutes to converge.

IPVS mode was the first answer: hash-based lookup instead of linear evaluation, so the data-plane cost is effectively constant, at the cost of a different set of edge cases. Since v1.33 there is a better one — nftables mode is GA, using a verdict map for O(1) service lookup and supporting incremental updates. It requires kernel 5.13+, and iptables remains the default for compatibility, so you have to opt in with --proxy-mode nftables.

From inside the pod. Uniform added latency to everything that goes through a Service IP, worsening as the cluster grows. Newly created Services taking a long time to become reachable — which reads as “DNS hasn’t propagated”, the single most misleading sentence in this whole space. It has not; the Service record was in CoreDNS immediately, and the datapath had not caught up.

The discriminating observation. Count the rules and time the sync:

sudo iptables-save | wc -l
sudo iptables-save -t nat | grep -c KUBE-SVC

# kube-proxy's own metrics, on the node
curl -s localhost:10249/metrics | grep -E 'sync_proxy_rules_duration_seconds|sync_proxy_rules_last_timestamp'

If sync_proxy_rules_duration_seconds is measured in seconds, the “DNS propagation delay” is a kube-proxy convergence delay. And the direct test: hit the endpoint pod IP rather than the Service IP. If the pod IP is fast and the Service IP is slow, the problem is in the proxy layer, and DNS resolved the name correctly both times.

The summary table

Impostor	Looks like	The one observation that separates it from DNS
Ephemeral port exhaustion	Timeouts above a throughput threshold, fine below	`EADDRNOTAVAIL` in the app; `TIME_WAIT` count vs `ip_local_port_range`. Destination-specific.
Conntrack table full	Node-wide packet loss, worsens over hours	`nf_conntrack_count` near `nf_conntrack_max`; `nf_conntrack: table full` in `dmesg`
Conntrack insertion race	Lookups that take exactly 5s, occasionally	`conntrack -S` → `insert_failed` climbing
MTU mismatch / PMTUD black hole	Handshake succeeds, then the connection hangs	A clean size threshold: `ping -M do -s N` passes at N, fails at N+1
NetworkPolicy / security group	Everything by name fails from this pod	Blast radius is this pod; an identical pod elsewhere works
CPU throttling on the caller	Client-side timeouts, server sees nothing wrong	`cpu.stat` → `nr_throttled` rising during the window
kube-proxy rule scaling	Uniform latency via Service IPs; slow “propagation”	Direct-to-pod-IP is fast, Service IP is slow; `sync_proxy_rules_duration_seconds`

Where they interact, which is why the DNS story is so tempting

None of these live in isolation, and two of them get worse under exactly the conditions where people reach for the DNS explanation.

Load couples them. Conntrack pressure rises with connection rate. The DNAT insertion race needs concurrency to fire at all — it is literally a race, so it is invisible at low QPS and unavoidable at high QPS. Port exhaustion is a rate problem by definition. CPU throttling bites hardest when the pod is busiest. So the incident always starts with “it’s fine in staging and broken in production at peak”, and every one of these explanations fits that sentence equally well.

MTU and conntrack compound. Retransmissions caused by an MTU black hole create more flows, which create more conntrack entries, which pushes a table that was at 70 % over the edge. Now you have two symptoms with one root cause, one of which (packet loss under load) looks like the classic conntrack story and one of which (size-dependent hangs) does not. Fixing the conntrack sizing makes the loss less bad and the real fault survives.

And DNS sits downstream of all of it. Name resolution is the first network operation almost any request performs, it is UDP, it is unretried-by-default within a tight timeout, and it fans out across several packets per logical lookup because of ndots. It is the canary. Killing canaries does not fix mines.

Part 2: a method, instead of a hypothesis

The method has one governing principle: eliminate layers in an order that makes each step cheap and each result unambiguous. Do not start where your intuition is strongest. Start where the answer is most binary.

Step 0: check the instrument before you check the network

I want to put this first because it cost me an hour and I think it is underrated.

I run a small two-node bare-metal cluster — Cilium in native routing mode, kube-proxy replaced. While bringing it up I wanted to confirm the pod-network MTU empirically rather than trusting the value Cilium had derived, so I did the obvious thing: shelled into a busybox pod and started binary-searching with ping -M do -s <size>.

It failed at every size. 1472 failed. 1400 failed. 100 failed. 1 failed.

Read as a network result, that is catastrophic — a total loss of pod-to-pod connectivity, except that everything else in the cluster was plainly working. Read correctly, it is not a network result at all: busybox’s ping applet does not implement -M. It was rejecting my command line, not my packets, and reporting failure in a form that looked exactly like the network being down.

Re-running from a nicolaka/netshoot pod, which carries the full iputils ping, gave the answer in about ten seconds: 1280 passes, 1281 fails. Exactly the MTU Cilium had configured, confirmed in one command.

The generalisable lesson is not “use netshoot”, though you should. It is this: a failure at every input size is a statement about your instrument, not about the system. Real network faults are almost always conditional — on size, on destination, on time, on load. When something fails uniformly and unconditionally, and other things are simultaneously working, suspect the measurement before you suspect the world. The same reasoning applies to nslookup in a distroless image, curl that is actually a busybox applet with different flag semantics, and a dig that is silently reading a different resolv.conf than the process you are debugging.

Minimum viable toolbox, which costs nothing to keep around:

kubectl run netshoot --rm -it --image=nicolaka/netshoot --restart=Never -- bash

# or attach to an existing pod's namespaces without changing it
kubectl debug -it <pod> --image=nicolaka/netshoot --target=<container> -n <ns>

kubectl debug with --target is the important form: it puts your tools inside the broken pod’s network namespace, which is the only place where the answers are valid.

Step 1: does the packet arrive at all?

Before reasoning about anything above L4, establish whether bytes are reaching the far end. This is the cheapest possible discriminator and it splits the search space in half.

Capture at both ends simultaneously:

# client side — inside the pod's netns
kubectl debug -it <client-pod> --image=nicolaka/netshoot --target=<ctr> -n <ns> \
  -- tcpdump -ni any -c 200 'host <server-ip> or port 53'

# server side — on the destination pod or node
tcpdump -ni any -c 200 'host <client-pod-ip>'

Four outcomes, four different investigations:

Client sends	Server receives	Server replies	Client receives	Conclusion
no	—	—	—	The problem is above the network. App, resolver config, or nothing was ever sent.
yes	no	—	—	Dropped in the path: policy, security group, routing, MTU, conntrack.
yes	yes	yes	no	Return path only. Asymmetric routing, SNAT/conntrack state, reverse-path filtering.
yes	yes	yes	yes	The bytes are fine. Look at latency, throttling, and the application.

That last row is the one people never check, and it is common. “The network is broken” very often means “the reply arrived and my process did not read it for 900 ms”.

On AWS, VPC Flow Logs answer the same question retrospectively and without a shell, which matters when the incident is over. ACCEPT on the way out and no matching record on the way in narrows it to the security-group and NACL layer immediately; REJECT names the layer for you.

Step 2: run two hypotheses in parallel, and kill one

This is the highest-value single step in the post, and it takes about five seconds.

Run these two commands at the same time, from the same pod:

# Hypothesis A: name resolution is broken.
dig +short +tries=1 +time=2 @<coredns-cluster-ip> <service>.<ns>.svc.cluster.local

# Hypothesis B: the network is broken, independent of names.
#   Note the hard-coded IP. No resolver is involved anywhere in this command.
nc -zv -w 3 <backend-pod-ip> <port>
# or
curl -sS -o /dev/null -w '%{http_code} %{time_connect}\n' --max-time 5 http://<ip>:<port>/healthz

Interpretation:

`dig` at resolver	Raw connect to IP	Conclusion
fails	fails	DNS is eliminated in one step. It is a path, policy or resource fault. Go to Step 3.
fails	works	A real DNS problem — or something that only breaks UDP/53. Go to Step 4.
works	fails	Not DNS. The name resolved; the connection did not. Path or backend.
works	works	Intermittent. Loop both for a few minutes and correlate with load.

The top-left cell is the whole point. Most “DNS incidents” I have watched people work on end there, ten minutes in, having already tried three CoreDNS changes. The raw-IP connect is what makes the elimination decisive: it is the only test in the toolbox that involves no name resolution at all, and therefore the only one whose failure cannot be blamed on DNS.

Two refinements worth having:

Query the CoreDNS pod IP as well as the Service IP. If the pod IP answers and the Service IP does not, CoreDNS is healthy and the problem is in the Service datapath — DNAT, conntrack, kube-proxy — which is a completely different investigation from “CoreDNS is broken”.
Try TCP: dig +tcp @<resolver> <name>. If UDP fails and TCP works, you are looking at a UDP-specific fault, which is the conntrack race, an MTU problem on large responses, or a policy that allowed TCP/53 and forgot UDP/53. That is a very strong signal and almost nobody checks it.

Step 3: check the resource layer before you blame the network

Three commands, each of which can end the investigation:

# a. Is the caller being throttled?  (inside the pod)
cat /sys/fs/cgroup/cpu.stat            # nr_throttled / nr_periods

# b. Is conntrack full, or racing?    (on the node)
cat /proc/sys/net/netfilter/nf_conntrack_count
sysctl net.netfilter.nf_conntrack_max
conntrack -S | tr ' ' '\n' | grep -E 'insert_failed|drop'
dmesg -T | grep -i conntrack | tail

# c. Are we out of ports or sockets?  (inside the pod, and on the node)
ss -s
ss -tan state time-wait | wc -l
cat /proc/sys/net/ipv4/ip_local_port_range

The ordering here is deliberate and it is about cost, not likelihood. These are read-only, take seconds, need no coordination with anyone, and each produces a number that is either obviously fine or obviously not. Compare that with the cost of the alternative — changing CoreDNS configuration on a live cluster, which requires a config change, a rollout, a waiting period to see whether the intermittent symptom recurs, and a rollback plan. You should not pay that cost until the cheap observations have failed to explain things. This is the whole of operational discipline in one sentence: order your diagnostics by (information gained) ÷ (cost and blast radius), not by which hypothesis feels most interesting.

Also check, in the same breath, whether CoreDNS itself is simply unhealthy — the most boring DNS failure and a genuinely common one:

kubectl -n kube-system get pods -l k8s-app=kube-dns
kubectl -n kube-system get pods -l k8s-app=kube-dns \
  -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.containerStatuses[0].restartCount}{"\t"}{.status.containerStatuses[0].lastState.terminated.reason}{"\n"}{end}'

A restartCount that is not zero with OOMKilled next to it is the answer, and it is found in one command.

Step 4: only now, go deep on DNS

If and only if Step 2 confirmed that names fail while raw connections succeed, the following are worth real time.

ndots:5 and search-domain amplification. The kubelet writes this into every pod’s /etc/resolv.conf under the default ClusterFirst policy:

nameserver 10.96.0.10
search <namespace>.svc.cluster.local svc.cluster.local cluster.local
options ndots:5

ndots:5 means: any name with fewer than five dots is treated as relative and tried against each search-list entry first, before being tried as absolute. That is deliberate — it is what makes kubectl exec ... curl http://payments work inside a namespace. It is also an amplifier for external names. Resolving api.stripe.com (two dots) produces:

api.stripe.com.<namespace>.svc.cluster.local   → NXDOMAIN
api.stripe.com.svc.cluster.local               → NXDOMAIN
api.stripe.com.cluster.local                   → NXDOMAIN
api.stripe.com                                 → answer

Four queries where one was needed — and since glibc requests A and AAAA, eight packets on the wire for one logical lookup. Add the node’s own search domains (a cloud-provider internal zone, typically) and it is worse. This is the mechanism behind “our DNS query rate is 8× what it should be”, and it is also why the conntrack race, which needs concurrency, fires so readily on DNS specifically.

Mitigations, in increasing order of blast radius:

A trailing dot. api.stripe.com. is absolute and skips the search list entirely. Free, precise, and requires editing application config. Worth doing for the two or three external hostnames that dominate your query volume.
Per-pod dnsConfig. Lower ndots for a specific workload that mostly talks externally:
```
spec:
  dnsConfig:
    options:
      - name: ndots
        value: "2"
```
Scoped to one workload, reversible, and the correct tool most of the time. The trade-off is about ordering, not about breaking things outright: at ndots:2, a name with two or more dots — payments.prod.svc — is tried as absolute first, so you pay a failed external lookup before the search list is applied as fallback. Short names (payments, payments.prod) still resolve on the first try. The cost is latency on the in-cluster names you shortened, plus one query per lookup escaping to your upstream resolver, which is worth thinking about if that upstream has a wildcard.
CoreDNS autopath. Makes the server infer the search path and answer the first query with a CNAME to the right answer, collapsing the fan-out without touching a single pod. The cost is that CoreDNS must watch all pods (pods verified in the kubernetes plugin), which raises its memory and API-server load, and it is the kind of clever that is hard to debug when it misfires. I would reach for dnsConfig first.
NodeLocal DNSCache. A DaemonSet caching resolver on a link-local address (169.254.20.10 by convention) that pods use instead of the Service IP. It is stable since v1.18. The reason it belongs in this post is stated explicitly in the Kubernetes docs: it skips iptables DNAT and connection tracking for the pod→resolver hop, which “will help reduce conntrack races and avoid UDP DNS entries filling up conntrack table”, and it upgrades the cache→CoreDNS hop to TCP, whose conntrack entries are freed on close rather than lingering for the 30-second UDP timeout. It is the structural fix for two of the impostors above, sold as a DNS feature. Note the operational tail: if the node-local-dns pod is OOMKilled it does not clean up its packet-filtering rules, and you get brief node-wide DNS downtime.

Retry storms. The default resolver timeout is 5 seconds with 2 attempts. Under partial packet loss, every client retries, doubling the query rate at exactly the moment the resolver is struggling. If your CoreDNS query-rate graph goes vertical during an incident, that is not the cause — it is the feedback loop. Look for what dropped the first packet.

CoreDNS cache and forward policy. forward defaults to policy random across the listed upstreams, with max_fails 2 and a health check every 0.5s; when all upstreams are marked down it assumes health-checking itself has failed and tries a random one anyway. Which brings me to my own case.

Part 3: the forward stanza that pointed nowhere

This happened on my own self-hosted cluster, and it is a good illustration of the method because I got it wrong first in exactly the way this post argues against.

The setup: a private zone for some homelab services, which I wanted resolvable from inside the cluster. The standard approach is a stub domain — a forward block in the CoreDNS Corefile sending that one zone to the resolver that is authoritative for it, and leaving everything else on the default path:

# Schematic. The resolver address is RFC 5737 documentation space, not the real one.
lab.internal:53 {
    errors
    cache 30
    forward . 192.0.2.53
}

The symptom, when it eventually mattered, was an AWX job doing a git sync: it hung and then failed on a timeout. Everything else in the cluster was fine. Other pods were fine. Other jobs were fine.

What I suspected and eliminated first. My first hypothesis was AWX. It is a big application with its own queues and a job-execution model that hides a lot, so a hang looked like an AWX problem, and I spent time in its logs. Eliminated when a plain git clone from a netshoot pod, of the same URL, hung the same way. Not AWX.

Second hypothesis: egress policy. A hung connection to an internal host is exactly what a missing egress rule looks like, and I had been tightening policies that week. Eliminated by Step 2 — the raw connect to the git host’s IP, with no name involved, connected immediately. So the path was open and the destination was up.

That single result also killed the third hypothesis before I formed it. If a hard-coded IP connects and the same request by name hangs, the fault is in resolution, and there is no point looking at MTU, conntrack, or anything else in the datapath. This is the case where DNS is the answer — and the value of running both tests in parallel is that it took one step to know that, rather than being where I started.

The confirming observation. With DNS established as the layer, the question narrowed to which part. The answer came from querying the resolver directly and watching the shape of the failure:

# in-cluster names: instant
dig +short kubernetes.default.svc.cluster.local

# external names: instant
dig +short example.com

# the private zone: hangs for the full timeout, then nothing
dig +time=2 +tries=1 git.lab.internal

Two zones instant, one zone hanging until timeout. A broken CoreDNS does not do that — it fails uniformly. A conntrack race does not do that — it is intermittent and zone-independent. A per-zone hang, reproducible on every attempt, points at exactly one thing: the stanza that handles that zone specifically. And a hang rather than an error means the upstream is not refusing or answering NXDOMAIN; it is not answering at all.

forward had been given an address the cluster could not reach. The zone’s real resolver lived on a network segment the nodes had no route to — correct from the machine where I had originally tested the address, unreachable from a pod. Every query for that zone went out, went nowhere, and burned the full timeout before failing. Everything else worked perfectly, which is why nothing was obviously “broken” for weeks: the cluster only touched that zone when something asked it to, and almost nothing did until the git sync.

The confirmation was one command from a pod, and it is the one I should have run first once DNS was the confirmed layer:

# can this cluster even reach the resolver it has been told to forward to?
dig +short +time=2 +tries=1 @192.0.2.53 git.lab.internal

Three things I took from it:

A forward target is a dependency, and nothing validates it. CoreDNS accepts any address. The Corefile is syntactically fine. There is no admission check, no startup probe, no warning. It fails only when someone asks for that zone, which may be weeks later, in a different context, to a different person.
The shape of a failure carries as much information as its content. “Hangs until timeout” versus “SERVFAIL immediately” versus “NXDOMAIN” are three different faults. The first means nothing answered. The second means something answered with an error. The third means something answered authoritatively that the name does not exist. I now read the failure mode before I read the logs.
Test connectivity from where the code runs. The address was verified from a laptop on a network that had a route. Pods are not on that network. This is the same class of mistake as a security group that works from the bastion.

Since then, any resolver I put in a forward block gets a check from inside the cluster, and CoreDNS gets errors and log enabled long enough to confirm the queries are going where I think they are.

Part 4: the case everyone cites — kubernetes#56903

Almost every “it’s always DNS” argument traces back, directly or through two or three intermediate blog posts, to this one issue. It is worth establishing what it actually was and, more importantly, what is still true, because it is now old enough to have been substantially fixed while its folklore continued unchanged.

What was reported. kubernetes/kubernetes#56903, “DNS intermittent delays of 5s”, opened 6 December 2017. DNS lookups from pods occasionally taking exactly five seconds. Reproducible on kops clusters on AWS. 273 comments over fourteen months.

What it turned out to be. Not DNS. The best technical write-up is Martynas Pumputis’s “Racy conntrack and DNS lookup timeouts” (August 2018), which traced it to races in netfilter’s connection tracking under DNAT. Because a Service ClusterIP is DNAT, and because glibc sends the A and AAAA queries in parallel from one socket, two UDP packets with identical tuples race through nf_conntrack_in and get_unique_tuple; neither finds a confirmed entry, both create one, one loses at insertion time and is dropped. The write-up identifies three variants of the race, differing in whether the two packets collide before confirmation, after confirmation, or get DNAT’d to two different backend endpoints. The resolver’s 5-second timeout supplies the characteristic latency. The observable fingerprint is the insert_failed counter in conntrack -S.

What was fixed, and where. Two upstream kernel patches:

Commit	Title	Landed
`ed07d9a`	netfilter: nf_conntrack: resolve clash for matching conntracks	Linux 4.19
`4e35c1c`	netfilter: nf_nat: skip nat clash resolution for same-origin entries	Linux 5.0

The first merges two clashing entries when their tuples match rather than dropping one. The second stops nf_conntrack_tuple_taken from treating a same-origin duplicate as a collision requiring a new source port. Both were backported into the stable series — 4.9 and 4.19 stable picked up the second in March 2019 — and distributions carried them separately, e.g. Ubuntu bug #1836816.

Separately, on the Kubernetes side, PR #78547 made kube-proxy emit -j MASQUERADE --random-fully in both iptables and IPVS modes where the local iptables supports it, shipping in v1.16. It is worth being precise about what that does, because it is routinely conflated with the DNAT race: --random-fully fully randomises source port selection during SNAT, which reduces a different clash — several simultaneous flows being masqueraded onto the same source port. It helps the general class of “simultaneous flows collide in NAT”. It is not the fix for the A/AAAA DNAT insertion race, and it does not remove the need for the kernel patches.

Issue #56903 was closed as completed on 22 February 2019.

What is still true in 2026, and what is folklore.

The kernel races described in 2018 are fixed in any kernel you should be running. 4.19 is from 2018; 5.0 from 2019. If you are on a supported distribution kernel today, you have both patches. Presenting the 5-second DNS timeout as a live, unavoidable Kubernetes hazard is repeating 2018. It is not one.
insert_failed is still a real counter and still worth watching. The specific races those patches addressed are gone; NAT is still a shared, concurrent, finite-resource data structure, and conntrack clashes have not been legislated out of existence. The counter costs nothing to scrape and tells you unambiguously whether you are in this family of problems.
Conntrack table exhaustion was never fixed and never will be, because it is not a bug. The table is finite by design. This is the part of the 2018 story that is still live, and it is the part people skip, because “the kernel had a race” is a better story than “you did not size a table”.
UDP DNS entries still occupy conntrack for nf_conntrack_udp_timeout (30s default) after a single query. Unchanged, and still the reason DNS-heavy workloads dominate conntrack occupancy.
The glibc parallel-A/AAAA behaviour is unchanged, so the concurrency that fed the race is still there; only the kernel’s handling of it improved. And the classic resolv.conf workarounds — options single-request-reopen, options use-vc — are glibc options. Alpine/musl images do not honour them, which is exactly why that advice kept failing for half the people in the issue thread (see the exchange at 3 September 2018). If you are shipping Alpine, that whole branch of the folklore does not apply to you.
NodeLocal DNSCache is stable and is still the right structural answer for DNS-heavy clusters — not because the race is back, but because bypassing DNAT and converting the upstream hop to TCP removes an entire category of failure rather than mitigating it.

So: cite #56903 as history that explains why everyone blames DNS, not as a diagnosis. If someone tells you your 2026 cluster has “the conntrack DNS bug”, ask them for conntrack -S. The counter is right there.

Part 5: the decision tree

For the case where someone says “there’s intermittent network weirdness” and you have five minutes. It is ordered by cost and decisiveness, not by likelihood.

Step 0 — is the instrument lying?

Fails at every size or every input, uniformly → suspect your tooling first. Re-run from netshoot, then continue.
Fails conditionally → continue.

Step 1 — run both probes at once. dig at the resolver, and a raw TCP connect to a hard-coded IP that involves no name lookup.

`dig`	raw connect	What it means
fails	fails	DNS eliminated. Path or resource fault — go to step 2.
fails	works	Genuine resolution fault — go to step 3.
works	fails	Name resolved, path did not. Backend or policy — go to step 4.
works	works	Not reproducing. Loop both under load and correlate.

Step 2 — path or resource.

Another pod on the same node works, only this one fails → NetworkPolicy, security group, or a sidecar.
Node-wide → check the cheap counters before touching the network:

Command	If it is rising	Conclusion
`cat /sys/fs/cgroup/cpu.stat`	`nr_throttled`	CPU throttling on the caller. Not the network.
`conntrack -S`	`insert_failed`, or count near max	Conntrack race or exhaustion. Size it, or add NodeLocal DNSCache.
`ss -s`	`TIME_WAIT` against `ip_local_port_range`	Port or pool exhaustion. Reuse connections, widen the range.
`ping -M do -s N`	a clean size threshold	MTU mismatch or PMTUD black hole. Fix MTU, allow ICMP type 3 code 4.

All four quiet → tcpdump at both ends and find where the packets stop.

Step 3 — genuine resolution fault.

dig at the CoreDNS pod IP works, only the Service IP fails → Service datapath, not CoreDNS. Look at DNAT and kube-proxy.
Pod IP fails too → ask which zones fail:
- one zone only → that zone’s forward stanza, and whether a pod can actually reach the upstream it names.
- external names only → upstream resolver, or ndots amplification and the retry storm it causes.
- everything → CoreDNS itself: restarts, OOMKills, replica count.

Step 4 — resolved but unreachable. Compare the Service IP against the endpoint pod IP directly, and check kube-proxy sync duration.

What I actually want you to take from this

DNS is one hypothesis. It should be tested, and the test costs one command run alongside another command.

The reason DNS gets blamed so disproportionately is not that people are careless. It is that DNS genuinely is where several lower-level faults become visible first. It is the first network operation almost any request makes; it is UDP, so it has no retransmission of its own; it runs against a tight timeout, so latency becomes failure quickly; and ndots fans one logical lookup out into several packets, so it samples the network more often than anything else you run. When conntrack drops packets, when the MTU is wrong, when a policy is too tight, DNS is the canary that stops singing first.

That is an argument for looking lower, not for looking at DNS first. The canary is telling you about the mine.

The practical version fits in three lines:

Before you form a hypothesis, run one command that involves no names at all. If it also fails, you have eliminated DNS in five seconds and saved yourself an hour.
Prefer the diagnostic that is read-only, node-local, and produces a number, over the one that requires changing a live cluster’s configuration and waiting to see if an intermittent symptom recurs. Order by information gained per unit of cost and risk.
When something fails at every input, check your instrument before you check the world.

And when it really is DNS — as it was, once, on my own cluster, with a forward stanza pointing at an address nothing could reach — you will have got there in one step instead of three, and you will know it rather than believing it.

Sources