Cluster autoscaler adds nodes during our morning ramp and for about 30-60 seconds afterwards we get a burst of resolution failures for in-cluster service names, mostly from pods that were just scheduled onto the new nodes. It clears on its own. CoreDNS is at two replicas, cluster is around 60 nodes at peak. Errors are name resolution timeouts rather than NXDOMAIN.
Timeouts rather than NXDOMAIN on brand new nodes points at the data path, not at CoreDNS's answers. Two things are almost certainly happening together.
First, kube-proxy on a new node has to program its rules before the ClusterIP for CoreDNS resolves to anything. Pods can be scheduled and start making DNS queries before that is complete, and you get exactly a 30-60 second window of timeouts.
Second, two CoreDNS replicas for 60 nodes is thin, and the morning ramp is when every new pod does its startup lookups at once. Run the cluster-proportional autoscaler for CoreDNS so replicas track node count, and spread them with a topology constraint so they are not both on the same node.
The change that will help most though is NodeLocal DNSCache. It puts a caching resolver on every node, so pods talk to something local over TCP to CoreDNS and stop depending on the ClusterIP path being ready. It also removes most of the conntrack UDP race problems people hit at this scale.
NodeLocal DNSCache plus proportional autoscaling of CoreDNS killed it entirely. Two weeks with no morning burst.
It is the single highest value DNS change most clusters can make, and it is a DaemonSet and a kubelet flag rather than a rearchitecture.
Longer term, retry logic in the applications matters here too. DNS is not a guaranteed-instant service and a client that gives up on one failed lookup during a scale event will keep finding ways to page you. Not an argument against fixing the cluster side, just the other half.
Worth confirming whether the failing pods are only the newly scheduled ones or also existing pods on old nodes. If existing pods are fine, it is startup ordering on the new node. If everyone gets it, CoreDNS itself is saturating and the replica count is the story.
@idempotent_ian · 5mo ago · 2 replies
Also look at ndots. The default
ndots:5means any name with fewer than five dots gets tried against every search domain first, so resolvingapi.example.comfrom a pod can be five queries before the one that works. During a burst that multiplies your query volume for no benefit. Using fully qualified names with a trailing dot for external hosts, or lowering ndots in dnsConfig, cuts the load substantially.Reply
Report
@csv_apologist · 5mo ago
Be a bit careful lowering ndots globally — short in-cluster names like
postgresrely on the search path. Setting it per workload is safer than cluster-wide.Reply
Report