Why can't I use a dashboard variable in an alert rule?

If you want one alert per environment, write the query with the value hard-coded and create one rule per environment. Tedious by hand, and the reason alert rules should live in configuration files rather than be clicked together — then it is a loop in a template and the tedium disappears.

If you want one alert that covers everything, drop the filter entirely and let the query return a series per environment. Alerting evaluates each returned series separately, so one rule produces one alert per environment automatically, and a new environment is covered the day it appears without anybody editing anything.

The second is almost always what people want once they see it. The variable was there to let a human narrow the view; an alert does not need to narrow, it needs to fire per thing.

The detail that makes it work is putting the identifying label into the alert's labels and message, so the notification says which environment rather than just that something is wrong.

The reasoning is straightforward once stated: a variable is a property of somebody looking at a dashboard, and an alert has nobody looking at it.

A dashboard variable is resolved from the state of the page — what you selected in a dropdown, possibly what is in the URL. An alert rule runs on a schedule in the background, with no browser, no session and no selection. There is no correct value to substitute, so the system refuses rather than guessing.

That also explains why it is not a limitation somebody will lift. It is not that the feature is missing; the input does not exist at evaluation time.

So the fix is to express the alert without needing a selection, and there are two ways to do that depending on what you actually want.

Broader habit that this points at: build alerts from the metric, not from the dashboard.

It is tempting to alert on a panel because the panel already looks right, and it produces alerts that inherit display choices — smoothing, a display window, a unit conversion — that were made for human eyes and are wrong for a threshold.

Writing the alert query separately, thinking about what the machine should measure, gives better alerts and stops the dashboard and the alert drifting apart when somebody edits a panel.

One thing worth checking before restructuring: whether the panel query is doing an aggregation that hides the label you need.

If the query sums across everything, the result has no environment label left, and the per-series alerting above cannot tell them apart. You need the grouping clause to keep the label you want to alert per.

This is the most common reason the multi-series approach produces one useless alert saying something somewhere is wrong. The fix is one clause in the query, and it is easy to miss because the dashboard looked right — the variable was doing the separating that the aggregation destroyed.