Ask
27
@k8s_kwame ·

Every kubectl command suddenly returns Unauthorized and nothing changed on the cluster

This morning every kubectl command against a cluster that was working yesterday returns an unauthorized error. No deployment, no upgrade, no configuration change that I am aware of, and the workloads themselves are still running fine.

So the control plane is up and serving, and it is specifically refusing me.

There is no obvious debug flag to get more detail out of the client. How do I work out what has actually gone wrong here?

5 answers Share
Report

Answering anonymously — a moderator will review it first.

  • @k8s_kwame · 16h ago

    "Nothing changed and it stopped working overnight" plus "the workloads are fine" is almost always an expired client certificate.

    Cluster certificates have a validity period, commonly a year for the ones minted at installation. Nothing changes on the day they expire except that time passes, which is exactly why it feels like nothing changed. And the workloads keep running because they are not authenticating as you.

    Check it directly. Pull the client certificate out of your kubeconfig, decode it, and read the validity dates:

    grep client-certificate-data ~/.kube/config | awk '{print $2}' | base64 -d | openssl x509 -noout -dates
    

    If the end date is in the past, that is your entire problem and the rest of the investigation is unnecessary.

    30
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
  • @k8s_kwame · 2d ago

    The other candidates, for completeness, since expiry is likely but not certain:

    • You are talking to the wrong cluster. A context switch that outlived its session, or a KUBECONFIG variable set in one shell and not another. Check kubectl config current-context before anything else, because this costs seconds.
    • The token expired if you authenticate with a service account token rather than a certificate.
    • An external auth plugin is failing — a cloud provider credential helper whose own login has lapsed. These often fail quietly and surface as a bare unauthorized.
    • Role bindings changed — but you would normally get forbidden rather than unauthorized. That distinction is genuinely useful: unauthorized means the server does not know who you are, forbidden means it knows and says no.

    That last distinction narrows the search more than anything else.

    21
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
  • @sysadmin_selin · 3d ago

    For the fix, and the important detail people get wrong: renewing the cluster certificates does not update your kubeconfig.

    The sequence on a self-managed cluster is usually:

    1. Renew the certificates on the control plane node using the cluster's own tooling.
    2. Restart the control plane components so they pick up the new certificates.
    3. Copy the freshly generated admin credentials from the control plane node into your own kubeconfig.

    Step 3 is the one that catches people. Your local file still contains the old certificate and key, so after a successful renewal you get exactly the same error and conclude the renewal failed. Replace the client-certificate-data and client-key-data values with the ones from the newly written admin config on the node.

    On managed clusters none of this applies — you re-run the provider's credential command and it writes a fresh kubeconfig for you.

    26
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
  • @packet_pavel · 19h ago · 2 replies

    On the debug flag: kubectl does have verbosity, it is -v with a number. kubectl get nodes -v=8 shows the full request and response including headers, which tells you which credential was actually presented and what the server said about it.

    At -v=9 you get request bodies too. It is a lot of output, but for authentication problems the useful part is right at the top — the TLS handshake and the identity being offered.

    15
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
    • @k8s_kwame · 2d ago

      Certificate expired eleven hours ago. The cluster was built exactly a year and a day before today. Slightly annoyed at how predictable that turned out to be.

      10
      Share
      Reply

      Answering anonymously — a moderator will review it first.

      Report