Priya

Question 3: temperature 0 is usually implemented as "skip sampling, take the highest-scoring token", so within one machine and one build it will normally repeat itself.

But it is worth separating deterministic from reproducible. Two things routinely break the second one even at temperature 0:

• Ties and near-ties. When the top two scores are equal to within floating point, which one wins can depend on summation order.
• Batching and hardware. Floating point addition is not associative, so the same maths executed in a different order — different batch size, different kernel, different GPU — can produce a slightly different score, and a slightly different score can flip a near-tie.

In practice you get identical output most of the time and an occasional divergence that looks like a ghost. If you need genuine reproducibility, pin the seed and the batch shape and the build, and do not treat temperature 0 as a substitute for any of that.

I would use the moment for the follow-up rather than closing it down, because the student has stumbled onto something real.

Ask them: given the mean is 1.5, how many households do you think have exactly one child? Let them guess, then show the distribution. Usually there is a large group at 0 and 2 and the mean is sitting in a gap.

That sets up the actual lesson — that the mean does not tell you the shape, and for skewed or lumpy data the median or the whole distribution says more. Income is the example everyone remembers afterwards: mean income and median income can be far apart, and which one gets quoted is often a rhetorical choice rather than a statistical one.

The student objected to a number that describes nobody. That is precisely the instinct you want to encourage, just aimed at the right target.

Your intuition that a too-flexible model is easier to pull back is actually sound, and it is worth keeping rather than discarding.

The standard workflow really is to build something with enough capacity to overfit and then constrain it — regularisation, early stopping, more data, fewer features. Deliberately overshooting and reining in is a normal and effective way to work, because it separates the question "can this model represent the pattern at all" from "is it generalising".

The distinction is between overfitting as a stage and overfitting as an outcome. Passing through it on the way, with held-out data telling you when to stop, is method. Shipping it is the failure. Your reasoning was right about the process and wrong about the destination.

There is an answer to question 2 that survives perfect arithmetic, and it is the one I would actually lead with.

Optimisation and calculus both prefer sums. Almost everything we do to a likelihood involves differentiating it, and the derivative of a product of a thousand terms is a nightmare of product rules, while the derivative of a sum of a thousand terms is a sum of a thousand derivatives. Each term depends on its own observation and nothing else, which is exactly what makes the gradient cheap and what makes it parallelise.

The same trick shows up wherever products of many things appear — path probabilities, sequence models, hidden Markov chains. Logs are not a numerical patch bolted on; they turn the problem into the shape that the tools were built for.

And for the exponential family — normals, Poissons, gammas — the log cancels the exp that is sitting in the density anyway, and you are often left with something polynomial. That is not a coincidence, it is why those distributions are convenient in the first place.

One caution on how confident to be, since you asked.

Each of these statistics is a single number and any one of them can land in an unlucky place. A real sequence occasionally has a short maximum run; a thoughtful faker who has read about this will deliberately insert a run of 7.

The robust approach is to check several independent-ish features and see whether they agree:

• longest run
• number of runs / alternation rate
• distribution of run lengths, which should be roughly geometric — about half the runs of length 1, a quarter of length 2, and so on
• counts of the four adjacent pairs HH, HT, TH, TT, which should be about equal

A genuine sequence passes all of them by construction. A fabricated one usually fails two or three, because the underlying bias — an aversion to repetition — shows up in every one of these views at once. If the two sequences disagree on only a single statistic, be honest that the evidence is weaker.

That separation is useful — I had the two objections mixed together and was dismissing both because the collinearity one clearly did not apply to me.

Small addition since it surprises people: the regression line does pass through the point (mean of x, mean of y), always. So it is genuinely centred in the cloud, just at a shallower angle than the cloud's spine.

Plot that point and it stops looking like the line has been pushed off to one side.

One more that catches people moving in the other direction — from machine learning to statistics, on the same data.

Collinearity is close to harmless for prediction and fatal for interpretation. Two nearly identical predictors will split the credit between them in an essentially arbitrary way; predictions barely move, individual coefficients swing wildly and can flip sign between refits.

This becomes a problem the moment somebody looks at your predictive model and asks "so which factor matters most?" — a question that quietly moves you from the framing where the assumptions did not matter to the one where they do. That transition usually happens in a meeting rather than in the code, which is why nobody notices it.

One clarification that trips people up when they first meet the definition: "linear" refers to the parameters, not to the shape of the curve.

Fitting y against x, x squared and x cubed is still linear regression, and still solved in closed form by least squares, even though the fitted curve is obviously not a straight line. What makes it linear is that y is a linear combination of the terms, whatever those terms are.

Whereas a * exp(b * x) is not linear regression, because b sits inside a function rather than multiplying a term. A curve can be bent and still linear; a straight-ish model can be nonlinear. The word is doing a technical job that its everyday meaning actively fights.

"The wrong models score 10^-340" is the sentence that fixed it. I was treating the likelihood as a goodness score on some absolute scale, and it never was one.