Kwame

"Nothing changed and it stopped working overnight" plus "the workloads are fine" is almost always an expired client certificate.

Cluster certificates have a validity period, commonly a year for the ones minted at installation. Nothing changes on the day they expire except that time passes, which is exactly why it feels like nothing changed. And the workloads keep running because they are not authenticating as you.

Check it directly. Pull the client certificate out of your kubeconfig, decode it, and read the validity dates:

grep client-certificate-data ~/.kube/config | awk '{print $2}' | base64 -d | openssl x509 -noout -dates

If the end date is in the past, that is your entire problem and the rest of the investigation is unnecessary.

Certificate expired eleven hours ago. The cluster was built exactly a year and a day before today. Slightly annoyed at how predictable that turned out to be.

The other candidates, for completeness, since expiry is likely but not certain:

• You are talking to the wrong cluster. A context switch that outlived its session, or a KUBECONFIG variable set in one shell and not another. Check kubectl config current-context before anything else, because this costs seconds.
• The token expired if you authenticate with a service account token rather than a certificate.
• An external auth plugin is failing — a cloud provider credential helper whose own login has lapsed. These often fail quietly and surface as a bare unauthorized.
• Role bindings changed — but you would normally get forbidden rather than unauthorized. That distinction is genuinely useful: unauthorized means the server does not know who you are, forbidden means it knows and says no.

That last distinction narrows the search more than anything else.

The first thing to know is the reason you could not find it: events are garbage collected after about an hour by default. By morning, the evidence of an overnight event is gone from the cluster entirely.

That is the single most important fact in this whole area, and it drives everything else. The cluster is not a record-keeping system; it is a control loop that knows the current state.

Where to look, in order of what survives:

• kubectl describe pod on a current pod shows recent events and, crucially, the last state of previously terminated containers with exit codes.
• kubectl get pod -o yaml shows lastState.terminated with a reason and exit code, which survives longer than the events do.
• The node's kernel log. If the machine ran out of memory, the kernel's out-of-memory killer logged it there, and that record persists.
• Your log aggregation, if you ship events and logs off-cluster. Which is the real answer to your question.

A pod recreated rather than restarted in place also points somewhere specific — that is the scheduler or a controller replacing it, not a container crashing.

To stop having this problem, the two changes worth making now rather than after the next occurrence:

Ship events somewhere durable. Whether that is your logging stack or a small exporter, it converts "the evidence expired" into "search yesterday at 3am". This is the highest-value hour of work in this whole area.

Set requests and limits deliberately. A great many mystery evictions are pods with no memory request being placed on a node that then runs out. Requests are what the scheduler uses to decide where things fit, and pods without them are the first to be evicted under pressure.

Also worth checking whether your nodes are being replaced on a schedule. Managed node pools with automatic upgrades will recreate pods overnight as designed, and it looks exactly like a mystery until you know it is happening.

This is a genuine version compatibility break rather than a misconfiguration, and knowing that saves a lot of wasted effort.

Kubernetes removed support for the older container runtime interface version at a specific release. From that release onward the kubelet speaks only the current API version. A container runtime that is older than that break implements only the previous one, so the two cannot talk at all — hence a failure at validation rather than a partial degradation.

So the mismatch is: your kubelet is newer than your container runtime. The fix is to upgrade the runtime to a version that implements the current API, not to change any kubelet setting.

The reason this catches people on a fresh node is that distribution repositories often carry an older runtime than the Kubernetes packages you added from a separate repository. Two package sources, two release cadences, and nothing warns you they have drifted apart.