Every kubectl command suddenly returns Unauthorized and nothing changed on the cluster
This morning every kubectl command against a cluster that was working yesterday returns an unauthorized error. No deployment, no upgrade, no configuration change that I am aware of, and the workloads themselves are still running fine.
So the control plane is up and serving, and it is specifically refusing me.
There is no obvious debug flag to get more detail out of the client. How do I work out what has actually gone wrong here?
@k8s_kwame · 14h ago
"Nothing changed and it stopped working overnight" plus "the workloads are fine" is almost always an expired client certificate.
Cluster certificates have a validity period, commonly a year for the ones minted at installation. Nothing changes on the day they expire except that time passes, which is exactly why it feels like nothing changed. And the workloads keep running because they are not authenticating as you.
Check it directly. Pull the client certificate out of your kubeconfig, decode it, and read the validity dates:
If the end date is in the past, that is your entire problem and the rest of the investigation is unnecessary.
Reply
Report