Debugging Playbook

First four commands, always

Before theorising, establish where the answer is coming from and whether the zone itself is right.

# 1. what does the system actually resolve (reads /etc/hosts and nsswitch)
getent hosts api.example.com

# 2. what does DNS say, and which resolver answered
dig api.example.com A

# 3. what does the authority say, bypassing every cache
dig @$(dig +short example.com NS | head -1) api.example.com A +norecurse

# 4. is the delegation intact from the root down
dig +trace api.example.com A

Step 1 versus step 2 is the highest-value comparison in DNS debugging. They disagree whenever /etc/hosts, nsswitch.conf, a container resolver, or a VPN is involved, and no amount of dig will reveal it.

Symptom to command to proof

Is it cached or is it wrong

This is the fork every DNS incident takes, and one comparison settles it.

# the cache your users hit
dig @8.8.8.8 api.example.com A +noall +answer

# the source of truth
for ns in $(dig +short example.com NS); do
  printf '%-28s %s\n' "$ns" \
    "$(dig @"$ns" +short api.example.com A +norecurse | tr '\n' ' ')"
done

# are the secondaries in sync
dig +nssearch example.com
host -C example.com

Different SOA serials across nameservers means a transfer is failing, and half your users get stale data.

Propagation check

There is no such thing as "DNS propagating". There is only your record's old TTL expiring in each resolver's cache, independently.

for r in 8.8.8.8 1.1.1.1 9.9.9.9 208.67.222.222 64.6.64.6; do
  printf '%-16s %s\n' "$r" \
    "$(dig @$r +short api.example.com A | tr '\n' ' ')"
done

8.8.8.8          198.51.100.30
1.1.1.1          198.51.100.30
9.9.9.9          198.51.100.20
208.67.222.222   198.51.100.30
64.6.64.6        198.51.100.30

One resolver behind means one cache has not expired. Nothing is broken and there is nothing to do but wait, or use the operator's purge form (dns.google/cache, one.one.one.one/purge-cache).

Global views: dnschecker.org, whatsmydns.net.

Clearing what you can clear

# Linux, systemd-resolved
sudo resolvectl flush-caches
resolvectl query api.example.com         # what resolved actually does

# macOS
sudo dscacheutil -flushcache
sudo killall -HUP mDNSResponder

# Windows
ipconfig /flushdns
ipconfig /displaydns | findstr example

# local unbound / bind
sudo unbound-control flush api.example.com
sudo rndc flushname api.example.com

The layers you cannot clear from a shell: the upstream recursive resolver, the browser's own cache (chrome://net-internals/#dns), a JVM caching addresses for the process lifetime, and any connection pool holding a resolved socket.

Container and Kubernetes specifics

kubectl exec -it POD -- cat /etc/resolv.conf      # the pod's resolver
kubectl exec -it POD -- nslookup kubernetes.default
kubectl exec -it POD -- getent hosts my-svc.my-ns.svc.cluster.local
kubectl -n kube-system logs -l k8s-app=kube-dns --tail=50

nameserver 10.96.0.10
search my-ns.svc.cluster.local svc.cluster.local cluster.local
options ndots:5

ndots:5 is the cause of most Kubernetes DNS latency. Any name with fewer than 5 dots is tried against every search-domain suffix first, so api.example.com (2 dots) costs four failed lookups before the real one. Fix it with a trailing dot (api.example.com.) or a per-pod dnsConfig lowering ndots.

Minimal images often have no dig, host, or nslookup at all. getent hosts is in libc, and nc -vz host port proves reachability without any DNS tool.

Escalation checklist

Work down, and stop at the first failure.

1. getent hosts NAME matches dig NAME +short? If not, it is the host, not DNS. 2. dig NAME returns NOERROR with answers? Note the RCODE if not. 3. dig NAME +cdflag changes a SERVFAIL to NOERROR? DNSSEC problem, go to the DS. 4. dig +trace NAME reaches an answer? If it stops, that hop is the delegation break. 5. Parent NS set matches the child NS set? dig NAME NS @<tld-ns> +norecurse. 6. Every listed NS answers with aa? Any that does not is lame. 7. All NS report the same SOA serial? dig +nssearch. 8. TTL on the record is what you set, and counting down? Then it is a cache. 9. Records themselves correct at the authority? dig @<auth-ns> ... +norecurse. 10. Still stuck: DNSViz for DNSSEC and the chain, Zonemaster for the full delegation audit.

Gotchas that waste the most time

- dig ignores /etc/hosts. Hours go into debugging DNS when a stale hosts entry was overriding everything. getent hosts first. - +short hides the RCODE. Empty output is NXDOMAIN, NODATA, or SERVFAIL, and they need completely different fixes. - Check the SERVER: footer. A VPN, a corporate resolver, or 127.0.0.53 answering is not the resolver you thought you tested. - Negative answers cache too. A name you just created can stay NXDOMAIN for the SOA minimum TTL, and nothing in the output says so beyond the SOA in the authority section. - The old TTL governs a change, not the new one. Lowering a TTL after publishing a change does nothing for resolvers that already cached. - A trailing dot changes the query. dig api.example.com may get search-list expanded, dig api.example.com. never does. - Browsers cache independently and ignore short TTLs. Chrome holds entries around 60 seconds regardless, so test with curl or dig, not a browser tab.