What is temperature actually doing to the model's output, and why does turning it up make things vaguer rather than just more random?

For question 2, the reason high temperature feels unfocused rather than differently focused is that it does not promote one alternative — it promotes the entire tail at once.

At any given position there are usually a handful of good continuations and a very long tail of tokens that are grammatical but wrong, plus a longer tail of tokens that are simply noise. Squashing the gaps hands probability to all of them together. Most of that mass lands in the tail because the tail is enormous, not because any individual tail token became attractive.

And the damage compounds. One odd token changes what the model is now continuing, so the next choice is made from a slightly worse position, and so on. That is why high-temperature text tends to drift rather than simply contain one surprising word: you are not sampling a different sentence, you are sampling a different sentence given the drift so far.

Which is also why top-p or top-k alongside temperature works so much better than temperature alone. Truncating the tail first means raising the temperature redistributes probability among candidates that were all plausible, and you get variety instead of decay.

Your instinct in question 1 is right and it is the useful place to start: the ranking never changes. Dividing every score by the same positive number cannot reorder them. The most likely token at temperature 0.1 is the most likely token at temperature 2.

What changes is the gaps between them.

The scores get exponentiated before they are normalised, and that is where dividing matters. Suppose two tokens have scores 6 and 4 — a gap of 2.

• Divide by 0.5 (low temperature): scores become 12 and 8, gap 4. After exponentiating, the first is about 55 times more likely.
• Divide by 2 (high temperature): scores become 3 and 2, gap 1. Now the first is only about 2.7 times more likely.

Same order, wildly different odds. Low temperature stretches the gaps until the top token is nearly certain; high temperature squashes them until the top ten are all roughly interchangeable.

The usual one-line summary is that temperature controls how much of the distribution's shape survives — and shape, not ranking, is what sampling actually consumes.

Cheap way to feel all of this rather than read it: take four or five numbers, divide them by T, exponentiate, and normalise, for T = 0.2, 1 and 5. Twenty seconds in any spreadsheet.

At 0.2 you get something that rounds to a one and a lot of zeroes. At 1 you get the shape the model actually produced. At 5 you get something close to a flat distribution, which is the real endpoint — infinite temperature means every token equally likely, which is to say, no model at all.

Seeing it once fixes the idea better than any explanation, mine included.