Managed cluster, three nodes of 4 vCPU and 16 GB, and I am trying to size the next pool honestly rather than by guessing. Between what the provider reserves and the DaemonSets we run for logging, metrics and CSI, a meaningful slice is gone before any of my workloads land. I want to know how to calculate it rather than discover it, and whether fewer bigger nodes is the obvious answer.
There are two separate deductions and people conflate them.
First, allocatable versus capacity. The provider reserves memory and CPU for the kubelet and the OS before anything of yours is schedulable. GKE publishes its formula and it is worth internalising because the shape is similar elsewhere: 25 percent of the first 4 GiB, 20 percent of the next 4, 10 percent of the next 8, 6 percent of the next 112, and 2 percent above that, plus another 100 MiB reserved for eviction handling. CPU is 6 percent of the first core, 1 percent of the second, 0.5 percent of the next two and 0.25 percent of everything above four.
Run that on a 16 GiB node and you lose roughly 1.8 GiB before you have deployed anything, which is most of your missing three.
Second, your own DaemonSets. Those come out of allocatable, on every node, and they are the part you control.
kubectl describe node shows you both: Capacity, Allocatable, and the Allocated resources block at the bottom.
The percentages are tiered specifically so that small nodes get hammered proportionally harder, which is the argument for bigger nodes in a nutshell. A 4 GiB node loses a quarter of its memory to reservation. A 64 GiB node loses well under a tenth.
The tiering was the piece I did not know. That plus our DaemonSet footprint accounts for essentially all of it.
kubectl describe node, look at Allocatable and then the Allocated resources table. That table is requests, not usage, and the scheduler only cares about requests. Most people asking this question are looking at usage graphs.
Measured our per-node tax so you have a shape to compare against. Logging agent about 300 MiB and a tenth of a core at our volume, node exporter tiny, kube-proxy tiny, CSI node plugin about 100 MiB, security agent about 250 MiB, service mesh sidecar injection is per pod rather than per node but adds up faster than any of these.
Call it half a core and just over a gigabyte per node before your workloads. On three 16 GiB nodes that is a bit over 3 GiB of overhead. On six 8 GiB nodes it is over 6 GiB for the same total memory. That is the real argument for consolidating.
Careful with fewer bigger nodes as a conclusion. Three things bite.
Blast radius: losing one node out of three when they are large means losing a third of your capacity at once, and the remaining two have to have headroom for it or nothing reschedules.
Pod density limits: the default cap is 110 pods per node and on AWS with the VPC CNI your real limit is often the number of IPs the instance type supports, which can be far lower than the memory would suggest.
Bin packing: one enormous node is worse at absorbing a burst of medium pods than two smaller ones in some scheduling patterns, especially with anti-affinity in play.
Mid-size and more of them has been the least painful setting for us.
Agreed on IP limits specifically, it catches people who sized on memory and then hit a ceiling they did not know existed. Prefix delegation raises it a lot if you are on that CNI and have not enabled it.
I sized a pool off usage graphs once and could not work out why half the nodes sat at 30 percent utilisation while pods went unschedulable. Every deployment had a request set to a round number somebody typed in 2021 and nobody had revisited. Fixing requests to something near reality freed more capacity than the node resize I was about to do, and it was free.
@leggy_seedling · 5mo ago
Add the control plane cost to your overhead maths if you are comparing to plain VMs. On EKS it is a flat 0.10 per cluster hour on a supported version, which is about 73 a month per cluster and goes to 0.60 an hour if you drift onto a version in extended support. That last part has surprised a lot of teams who thought upgrades were optional.
Reply
Report