Dana

The layers, in the order I would run them:

1. Ask the package manager first. Whatever your distribution uses, query for the package by name. This catches anything installed the supported way and takes seconds. It also tells you the version, which the filesystem alone will not.

2. Search the filesystem for jar files matching the library name. Update your file index first if you are using an indexed search, otherwise you are querying yesterday's disk. Expect this to find vendor-bundled copies the package manager missed.

3. Search inside archives. This is the step people skip and it is where the real findings are. There are purpose-built scanners for exactly this problem that walk nested archives and check the classes inside rather than the filenames outside. Use one rather than writing your own — the nesting gets deep and the edge cases are numerous.

4. Ask what is actually running. List the running Java processes and look at their classpaths. A vulnerable jar sitting on disk that nothing loads is a lower priority than one in a live process.

Worth knowing there are usually three possible managers on a server, and which one you have determines the answer:

• systemd-resolved — the stub listener case above. Configure in resolved.conf.
• NetworkManager — common on desktops and some server images. It writes resolv.conf from the connection profile, so configure the connection, not the file.
• The DHCP client — on a plain server image with neither of the above, the DHCP client overwrites resolv.conf on every lease renewal with whatever the network handed you. That is why your edits survive for a while and then vanish.

The quickest way to tell which you are dealing with: look at the first line of /etc/resolv.conf. It almost always contains a comment naming the program that generated it. That one line saves a lot of guessing.

Two adjacent things that bite in the same situation.

Ciphers and MACs go the same way. The removals happen in waves across releases, so a machine that connected to your switch last year may refuse this year after an upgrade, with a different algorithm named each time. If you maintain old equipment, expect this as a recurring maintenance item rather than a one-off.

Distribution-level crypto policies can override your config. Several distributions now ship a system-wide policy that constrains what SSH will do regardless of per-host settings. If your host block appears to be ignored, that is usually why, and the fix is at the policy level rather than in your config file.

The long-term answer for a switch is firmware, if any exists. Where it does not, the config entry is a legitimate way to keep managing hardware that outlived its crypto.

One quick check that often resolves it immediately: compare the number of nodes now with what you expect, and look at node ages.

If a node is two hours old and the recreated pods are two hours old, the pods did not do anything — the node was replaced underneath them. That is a five second check and it eliminates the entire application-level investigation.

The general lesson for anyone building nodes: pin the versions of the runtime and the Kubernetes components together, and treat them as one unit.

Kubernetes publishes a supported version skew for each release. Installing the kubelet from one repository and the runtime from another, both floating on latest, guarantees you will eventually meet a break like this one — usually during an unattended upgrade, at night, on a node that then fails to rejoin.

Hold the packages at chosen versions and upgrade them deliberately as a set. It is more work per upgrade and dramatically less work per incident.

A variant worth checking: is it everything failing, or only third-party repositories?

If the distribution's own repositories work and only added ones fail, the trust store is fine and the problem is those repositories' certificates — expired, misconfigured, or signed by an authority that got distrusted. In that case the fix is per-repository and reinstalling CA certificates achieves nothing.

Quick way to tell them apart: try fetching a distribution URL and a third-party one with a plain HTTPS client and compare the errors. Different failure messages mean different problems, and it saves you performing surgery on a healthy trust store.

It is a bigger event, and the answer is almost certainly that the release reached end of life and the project retired the infrastructure that served it.

The mirror list service exists to hand your package manager a set of nearby mirrors that carry current packages for a supported release. Once a release is no longer supported, there are no current packages, mirrors stop carrying it, and the service that points at those mirrors has nothing to point at. Removing the DNS record is the last step of a decommissioning that happened on a published schedule.

So nothing broke on your side and nothing broke on theirs. Your server outlived the support window and found out via DNS, which is an abrupt way to be told.

A 404 rather than a connection error is the whole diagnosis: the mirror is fine, and the directory for your release is no longer on it.

That happens when a release reaches end of life. Interim releases have short support windows — often around nine months — and when one lapses, its files are removed from the general mirror network and moved to an archive host. The mirror keeps serving every supported release perfectly while returning not-found for yours.

So nothing broke. Your release retired on a published date and the mirrors implemented that.

The immediate fix is to repoint your sources at the archive host that holds retired releases, which restores your ability to install packages from the frozen final state. The real fix is to upgrade to a supported release, because the archive receives no security updates and never will.