Six node pool, all the same instance type at 4 vCPU. Deployment has ten replicas, nine are Running and one has been Pending since last night. Events say 0/6 nodes are available with a string of reasons I have read twenty times. There is a topology spread constraint and a nodeSelector on the pod spec, both inherited from a template I did not write. Cluster autoscaler is on and has not added a node.
I had the identical symptom and it was the spread constraint. whenUnsatisfiable set to DoNotSchedule with maxSkew 1 across six nodes and ten replicas: ten does not divide by six, so once the first nine are placed the tenth has nowhere legal to go and it waits forever rather than breaking the skew rule. Switching to ScheduleAnyway fixed it in one line, at the cost of a slightly uneven distribution I did not care about. Worth checking whether your replica count and node count are coprime before you look at anything else.
Before you tune anything else, check the arithmetic on requests against allocatable rather than capacity. A pod requesting 4 CPU will never schedule onto a 4 vCPU node, because the kubelet and the OS reservation mean allocatable is always meaningfully less than capacity. I have watched two different teams lose an afternoon to a request that exactly matched the instance size and looked like it should just fit.
On the autoscaler not scaling: it will not add a node if the pending pod would not fit on a node of the shape it can add, and it does not simulate constraints you might expect it to. A pod that fails a nodeSelector for a label that only exists on a pool at max size will sit there forever while the autoscaler logs that it has nothing useful to do. Its logs are verbose and unusually honest, read them.
Pushing back on the just lower the request advice that always turns up in these threads. Lowering requests until it schedules is how you end up with a cluster that schedules everything and performs terribly under load, and then somebody adds limits on top and you get throttling nobody can explain. Right-size from observed usage with headroom, then fix the scheduling constraint that is actually blocking you. They are separate problems and merging them makes both worse.
Strong agree. The other version of this is deleting the topology spread constraint entirely because it was in the way, then discovering during a zone outage exactly why somebody put it there.
My version of this was hard pod anti-affinity with more replicas than nodes, which is the same arithmetic trap as yours. Six nodes, eight replicas, requiredDuringScheduling. Two pods Pending permanently and a very confused me. Preferred rather than required is right for almost everything that is not a database.
@crashloop_cara · 3mo ago · 3 replies
Read the event string literally, phrase by phrase, because each clause is a different predicate failing on a different subset of nodes and they add up to six.
Insufficient cpu or memory means the requests do not fit in what is left. node(s) had untolerated taint means a taint. node(s) didn't match Pod's node affinity/selector means your nodeSelector. didn't match pod topology spread constraints means the skew rule. node(s) had volume node affinity conflict means the PV is pinned to a zone.
The sum across clauses will equal your node count. If two clauses account for five nodes and one for the last, you have two separate problems, which is why nine scheduled and one did not.
Reply
Report
@first_dollar_fi · 3mo ago
kubectl get events --field-selector involvedObject.name=POD sorted by timestamp is worth it too, because the scheduler updates the message as conditions change and the current one may not be the original reason.
Reply
Report
@trace_tilly · 3mo ago
Adding them up was the trick. Five nodes fail the spread constraint and one fails on cpu. Two problems wearing one trench coat.
Reply
Report