Ask
26

Every datasource says "Data source not found" after a minor upgrade, and the server has no internet — what changed?

We upgraded our dashboard server by one minor version and every one of our search-backend datasources stopped working. The panels all show "Data source not found".

Looking at what the browser is actually requesting, there is a call to a plugin settings endpoint that comes back with a message saying no installed plugin has that id. But the CLI's plugin list does show it, at the version from before the upgrade.

It affects the ones configured by hand in the UI and the ones we create through provisioning files equally, so it is not a config problem on our side.

The other thing in the log, which I suspect is the whole story: on startup something tries to install a plugin and fails to reach a vendor URL. This machine has no route to the internet and is not going to get one. Auto-updates are already disabled and it tries anyway.

What actually changed here, and what is the correct fix for a machine that cannot phone home?

4 answers Share
Report

Answering anonymously — a moderator will review it first.

  • @core_plugin_split · 4h ago

    Your suspicion is right and the log line is the whole story.

    What changed. A datasource that used to be built in has been moved to being a separately distributed plugin. Functionally it is the same thing. Operationally it is completely different: something that shipped inside the package now has to be fetched, and on first start after the upgrade the server tries to fetch it.

    That is why the two things you found look contradictory but are not. The CLI lists the plugin because the old bundled copy is still on disk from before the upgrade. The running server does not recognise it, because it now expects it in the location and shape that installed plugins use. So it is present and unusable at the same time, and "no installed plugin with that id" is literally accurate from where the server is standing.

    Why disabling auto-update did not help. Auto-update governs upgrading plugins you have. This is a first install of something the server now considers missing, which is a different code path and is not what that setting controls. Worth knowing generally: "do not update" and "do not install" are usually separate switches, and operators reach for the first one expecting it to cover both.

    Why your provisioned datasources broke too. Provisioning creates the datasource record, and the record names a plugin that has to exist for it to resolve. No plugin, no resolution, regardless of how the record got there. Provisioning is not a substitute for the plugin being installed and never was — it simply used to be, because the plugin was always there.

    So nothing on your side is misconfigured. The upgrade added a dependency on network access that your server is designed not to have.

    30
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
  • @offline_install_ida · 4h ago

    The fix on an isolated machine is to put the plugin there yourself, and there are two ways depending on how much you want to automate.

    By hand. Download the plugin archive on a machine that does have internet, move it across, and unpack it into the plugins directory the server is configured to read. Match ownership and permissions to the rest of that directory, then restart. The plugin list should then show it as a normal installed plugin rather than the bundled leftover, and the datasources resolve without touching their configuration.

    Via the CLI, pointed at a local file or an internal mirror. The plugin install command accepts a URL rather than only a name, so an internal file server or artifact repository works and gives you something repeatable. This is the better answer if you have more than one server, because the manual route is one you will get wrong on the third machine.

    Two things to check while you are in there:

    • Version compatibility. You want the plugin build that matches the server version you upgraded to, not the newest one available. This is exactly the situation where the newest is not the right answer.
    • Whether the old bundled copy is still lying around. Having both can produce confusing results, and cleaning it up after the real one is in place removes a whole category of future puzzlement.

    Longer term, the general lesson for an isolated network: assume anything that used to be bundled can stop being bundled. Mirror the plugins you depend on to somewhere inside your perimeter and install from there as a matter of course, so a packaging decision made upstream is no longer able to take your dashboards down on a Tuesday morning.

    25
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
  • @postinst_trap_pelin · 4h ago

    A warning about the neighbouring problem, because people arrive at this from the other direction and it is genuinely worse.

    On distributions using system packages, an upgrade around this change can fail during package configuration — the post-install step tries to move the bundled plugins directory aside, finds it is not empty, and stops. The package is then left half-configured, and because package managers process everything as a batch, the whole upgrade run is now wedged and refuses to proceed until it is resolved.

    The thing not to do is what the error tempts you into: editing the maintainer script to skip the failing line. It gets you past the immediate error, and it does it by not performing a step the package thought was necessary, on a machine you now cannot reason about. Whatever the next upgrade assumes about that directory is now untrue.

    The safer shape is to sort out the directory the script is complaining about first, then let the original script run. Take a copy of it somewhere else before you touch anything, so "put it back" stays available.

    And the general habit this argues for: snapshot before a minor upgrade of anything that owns its own plugin directory. Not because minor upgrades are risky in themselves, but because packaging changes hide inside them, and a packaging change is the kind that leaves the machine in a state no runbook covers.

    21
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
  • @core_plugin_split · 4h ago

    One note for anyone reading this later on a machine that does have internet and wondering whether it applies to them.

    It still does, just quietly. With a route out, the missing plugin is fetched on first start and everything works, so the change passes unnoticed — which means you have silently acquired a startup dependency on a vendor endpoint being reachable. That is fine right up until the morning it is not, and then a dashboard server that has run for two years without incident fails to come up properly and nothing in your own change history explains it.

    Worth knowing which of your plugins are now fetched rather than bundled, even if you never do anything about it. The list is short and the knowledge costs nothing until the day it is the only thing that helps.

    12
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report