A container got OOMKilled with a 512Mi limit, exit code 137. kubectl top pod had been showing it steady around 300Mi for the previous ten minutes, and our dashboard agrees. Nothing in the app logs suggests a spike. I am new to running Kubernetes and I cannot work out whether the metrics are lying, the limit is wrong, or I am reading the wrong number entirely.
Worth knowing the accounting rule too: the cgroup counts page cache against your limit, so a container that reads or writes large files can be killed even though the application's own allocations are modest. If your workload touches big files, that is a real and very confusing source of OOMKills.
This one cost me a day. A log shipper reading rotated files was being counted against a limit I had sized from application heap alone.
Also confirm which container was killed. kubectl describe pod gives last state per container, and in a pod with sidecars people routinely assume it was the main application when it was actually a proxy or a log agent with a much smaller limit.
Practical habit that saves time later: set requests from observed steady state and limits from observed peaks, and only after you have watched a workload through a full weekly cycle. Guessing round numbers on day one is how everyone gets here.
Simplest answer is to remove the memory limit entirely, then it cannot be OOMKilled.
That moves the problem rather than removing it. Without a limit the container can consume the node's memory and the kernel starts killing whatever it likes on that node, which is a much worse day than one restarting pod.
@idempotent_ian · 3mo ago · 3 replies
Both numbers are honest, they are just measuring different things at different resolutions.
kubectl topreads from metrics-server, which scrapes on an interval — typically 15 to 60 seconds — and shows working set at the moment of the scrape. A container can allocate hundreds of megabytes and get killed in well under a second, and no scrape ever sees it. Steady at 300Mi on a one minute average is entirely compatible with a 250Mi spike that lasted 400 milliseconds.The kernel, on the other hand, checks against the cgroup limit on every allocation. It does not average anything.
What to do about it: look at
container_memory_working_set_bytesat the finest resolution your monitoring keeps, and checkkubectl describe podfor the last state and the restart count to confirm the timing. Then think about what in that container allocates in bursts — decoding a large request body, loading a file, a batch job, or a garbage collector that has been given a heap size close to the container limit.Reply
Report
@pentalobe_junie · 3mo ago
It is a JVM with a max heap set to 512Mi inside a 512Mi container, which after reading this is obviously wrong. Thank you for explaining the sampling part rather than just telling me to raise the limit.
Reply
Report
@csv_apologist · 3mo ago
That is the single most common version of this. The JVM needs headroom above the heap for metaspace, thread stacks and native buffers — as a rule of thumb leave 25% of the container limit outside the heap.
Reply
Report