Ruth

For question 2, the reason high temperature feels unfocused rather than differently focused is that it does not promote one alternative — it promotes the entire tail at once.

At any given position there are usually a handful of good continuations and a very long tail of tokens that are grammatical but wrong, plus a longer tail of tokens that are simply noise. Squashing the gaps hands probability to all of them together. Most of that mass lands in the tail because the tail is enormous, not because any individual tail token became attractive.

And the damage compounds. One odd token changes what the model is now continuing, so the next choice is made from a slightly worse position, and so on. That is why high-temperature text tends to drift rather than simply contain one surprising word: you are not sampling a different sentence, you are sampling a different sentence given the drift so far.

Which is also why top-p or top-k alongside temperature works so much better than temperature alone. Truncating the tail first means raising the temperature redistributes probability among candidates that were all plausible, and you get variety instead of decay.

The other half is that they are not equally visible, and in practice that is what does the damage.

An underfitted model announces itself. Training error is bad, validation error is bad, everyone in the room can see the model is not good enough, and nobody ships it by accident.

An overfitted model looks superb right up until it meets real data. If your validation split is not clean — leakage, duplicated rows, a time series split randomly instead of chronologically — it can keep looking superb through evaluation too. The failure then surfaces after deployment, which is the most expensive place to discover anything.

A failure you can see is cheaper than a failure you cannot, even when the second one has a better number attached.

Range first, because it is short: log of 1 is 0, and log of anything smaller is negative, running down towards minus infinity as the probability approaches 0. So log probabilities live in (-infinity, 0], and closer to zero is more likely.

That is worth saying out loud because it flips the direction people expect. -2 is a much better score than -8420. When you see a log-likelihood of -8420 your reaction should not be "that is a huge negative number, something is broken" — it should be "that is a number I can only interpret by comparing it to another one from the same data."

Which is the real habit to build: log probabilities are almost never meaningful alone. They are meaningful as differences. A difference of 2 in log space means one thing is about 7.4 times more likely than the other, and that ratio is the thing you actually care about.

If you want something you can put a number on rather than an eyeballed comparison, count the runs and use the fact that the total number of runs has a known distribution.

With n flips split roughly evenly, the expected number of runs is about n/2 + 1, so around 101 for 200 flips, with a standard deviation of about 7. A fabricated sequence with too much alternation will show up as too many runs — 120 or more is roughly three standard deviations out and effectively settles it.

That gives you the justification your colleague asked for: not "this one looks faked" but "this sequence has 124 runs where 101 ± 7 is expected, and the other has 98".

There is a real problem inside the range, and it is not overfitting in the usual sense. It is that polynomial terms are not local.

A degree-9 term is a single global function evaluated across your whole range. That means a cluster of points at the far right of your data influences the fitted shape at the far left, because both are being explained by the same coefficient on x^9. Move one outlying point at one end and the curve wobbles somewhere else entirely.

That is almost never what you want. When we fit a curve we usually believe something like "the relationship is smooth" — a local claim. Polynomials deliver "the relationship is one specific global shape", which is a much stronger claim that nobody meant to make.

The practical consequence is that the fit is unstable in a way that cross-validation on a single split will not necessarily reveal. Refit on a bootstrap resample a few times and watch the middle of the curve move.

Your eyes are fine and so is the regression. They are measuring different distances.

Ordinary least squares minimises the vertical distance from each point to the line — the error in y only, with x treated as known. Your eye, and the principal axis, minimise perpendicular distance, which treats the two directions symmetrically.

Those give different lines whenever the data is noisy, and the regression line is always the shallower of the two. That is not a defect; it follows from the question being asked. "Given this x, what is my best guess for y" is not symmetric in x and y, so its answer should not be either.

The visual check that makes this click: draw the vertical segment from each point to the line rather than the perpendicular one. Under that view the regression line is obviously centred and the principal axis obviously is not.

Neither course is sloppy. They are answering different questions, and the assumptions belong to one of the questions and not the other.

The statistics framing is aimed at inference: is this coefficient different from zero, how wide is the interval around it, can I say this predictor matters. Every one of those outputs is a probability statement, and a probability statement needs a probability model. The normality and constant-variance conditions are what license the standard errors, the p-values and the confidence intervals. Break them and those numbers are still printed, but they are wrong.

The machine learning framing is aimed at prediction: how close is the predicted value to the real one on data the model has not seen. That question has an empirical answer — hold data out and measure. You are not making a probability claim about a coefficient, so you do not need the machinery that would justify one.

So the honest version is: the assumptions are conditions for the inferential outputs, not for the fitting. Drop the outputs and you drop the conditions with them.

They are two different kinds of thing, and confusing them is common enough that it is worth being precise once.

• Linear regression names the model: you are asserting that the expected value of y is a linear combination of your predictors. That is a claim about shape.
• Least squares names the fitting criterion: whatever the model, choose the parameters that minimise the sum of squared residuals. That is a claim about what counts as a good fit.

A model needs a criterion and a criterion needs a model, so in ordinary use you pick both and nobody distinguishes them. They coincide so reliably in introductory material that the two words become one idea.

They come apart in both directions, which is the cleanest way to see they are separate.

The clarifying experiment is two minutes and it makes the point better than an explanation.

Compute the likelihood of your data under the true parameters, and then under parameters that are noticeably wrong — mean shifted by two, say. Both numbers will be unreadably small. Now take the ratio, or equivalently the difference of the log-likelihoods.

You will find the true parameters beat the wrong ones by an enormous factor. That factor is the entire content of the method. Maximum likelihood never asks "is this number big" — it asks "which parameter values make this number biggest", and the answer to that is perfectly stable regardless of how small everything is in absolute terms.