Lorenzo

The other reliable move is analogy to a number they already accept without complaint.

A batting average, a goals-per-game figure, a grade point average of 3.4 when no single course awards a 3.4, a household with 2.3 cars. They have lived comfortably with all of these for years. Nobody has ever objected that you cannot score 1.7 goals.

That usually gets a laugh and does real work, because it relocates the confusion. The problem is not the arithmetic and never was — it is that "children" feels like it should be a whole thing in a way "goals per game" does not. Naming that is more useful than defending the mean, and it opens the genuinely interesting question underneath: when should you distrust an average?

There is a version of your friend's instinct that is right, and it might be worth conceding at the table so the argument ends properly.

If you want the probability of the specific story — "he will say 17 and then roll 17" — evaluated before he speaks, that genuinely is 1 in 400 (assuming his guesses are uniform). Told that way it sounds far more impressive.

This is the everyday version of a real trap in statistics: choosing which event counts as remarkable after seeing it. The event "he called it" is common. The event "he called 17 specifically" is rare, but it only became a candidate for remarking on because it happened. Twenty different rare events would each have felt equally striking.

So: 1 in 20 for what actually impressed you, 1 in 400 for a prediction nobody made.

Your instinct in question 1 is right and it is the useful place to start: the ranking never changes. Dividing every score by the same positive number cannot reorder them. The most likely token at temperature 0.1 is the most likely token at temperature 2.

What changes is the gaps between them.

The scores get exponentiated before they are normalised, and that is where dividing matters. Suppose two tokens have scores 6 and 4 — a gap of 2.

• Divide by 0.5 (low temperature): scores become 12 and 8, gap 4. After exponentiating, the first is about 55 times more likely.
• Divide by 2 (high temperature): scores become 3 and 2, gap 1. Now the first is only about 2.7 times more likely.

Same order, wildly different odds. Low temperature stretches the gaps until the top token is nearly certain; high temperature squashes them until the top ten are all roughly interchangeable.

The usual one-line summary is that temperature controls how much of the distribution's shape survives — and shape, not ranking, is what sampling actually consumes.

For question 2 — the numerical argument is real and it is not marginal.

Multiply a thousand probabilities of 0.5 together and you have 0.5^1000, which is around 10^-301. That is close enough to the floor of double precision that a few thousand more observations underflow to exactly 0. And once it is 0, it is not "very small", it is gone: you cannot divide by it, you cannot compare two of them, and every ratio you were about to compute is 0/0.

In logs the same quantity is a sum of a thousand copies of about -0.693, which is around -693. Perfectly ordinary number, no drama.

So yes, mostly numerical — but the reason it bites so fast is worth internalising. Likelihood of a dataset is a product over observations, so the exponent grows linearly with your sample size. Underflow is not an edge case you might hit; with any real dataset it is the default outcome.

The other half of the standard advice, which usually goes unsaid: there is a better tool for the same job, so the trade-off is not "polynomial or nothing".

Regression splines and natural splines give you the flexibility without the globalness. They are piecewise polynomials joined smoothly at knots, so a point on the right only affects the fit near the right. Natural splines go further and constrain the fit to be linear beyond the outermost knots, which kills the edge behaviour by construction.

You get roughly the same expressive power for the same number of parameters, with local influence and far more stable coefficients. Once that alternative exists, "do not use degree 9" stops being a warning about danger and becomes ordinary advice about picking the better of two tools.

If you want to keep it simpler still, a moving-window smoother will answer most "what does the shape look like" questions without committing to a parametric form at all.

The consequence worth knowing is that the two lines answer different practical questions, so which one you want depends on what you are going to do with it.

• Predicting y from x — measurement, forecasting, any "given this input, estimate this output" task. Ordinary least squares, vertical distances, and the shallower line is genuinely the better predictor. Its shallowness is regression to the mean doing its job.
• Estimating an underlying relationship where both variables are measured with error — two instruments measuring the same thing, calibration between methods. Then treating x as known is a lie, and the perpendicular fit is closer to what you mean.

A useful sanity check for the first case: regress y on x, then regress x on y, and plot both. You get two different lines, and neither is the principal axis. If those two lines are far apart your data is noisy enough that the distinction matters; if they nearly coincide it does not.

That reframing plus "independence is not optional either way" is what I was missing. I had assumed the machine learning course dropped all four, when actually it dropped the two that only support inference.

Concrete examples in both directions, since that is what you asked for.

Linear regression fitted by something other than least squares. Fit the same linear model by minimising the sum of absolute residuals instead of squared ones. You get a valid linear regression — same model, same interpretation of coefficients — with a different line, one that is far less sensitive to outliers. Quantile regression is the same idea generalised, and ridge and lasso are linear regression with a penalty added to the criterion.

Least squares applied to something that is not linear regression. Fit a curve like a * exp(b * x) by minimising squared residuals. That is nonlinear least squares: the criterion is unchanged, the model is not linear in its parameters, and you need an iterative solver rather than a closed-form solution.

Once you have seen both, the pairing stops looking like one thing.

You are reading it as an absolute score when it is only ever a relative one. Two things are making it tiny, and neither says anything is wrong.

It is a product over every observation. Each data point contributes a factor less than 1, so with 500 points you are multiplying 500 small numbers. 10^-300 for a few hundred observations is completely ordinary — the exponent scales with sample size, so the number carries no information about fit on its own.

For continuous data it is a density, not a probability. The probability of drawing any exact real value is zero. What you computed is a density height, which is only meaningful multiplied by an interval width. It can even exceed 1 for a sufficiently narrow distribution, which is a good sign that it is not a probability.

So the correct answer to "how can 10^-300 be the right model" is that the wrong models score 10^-340, and you would only ever have looked at the ratio.