Omar

Concede first is the key. Every year I have watched people lose the room by defending the number, when the student's objection is not to the number at all — it is to being told that a summary is a description of an individual.

It breaks down on the phrase "does well somewhere". Training performance is not somewhere. It is the one place where the score is guaranteed not to mean anything, because that is the data the parameters were chosen to fit.

The asymmetry that matters is about how bad each failure can get:

• Underfitting has a floor. A model that is too simple converges on the average behaviour. It will be consistently mediocre and roughly as mediocre on new data as on old.
• Overfitting has no floor. A sufficiently flexible model fitted to noise can produce predictions that are arbitrarily wrong on new data — not slightly worse than the mean, but far worse than predicting the mean.

So they are not two symmetrical errors. One is bounded and the other is not, and unbounded downside is the thing you protect against first.

Add to that: the interior misbehaviour is real, not only the edges.

The classic demonstration is fitting a high-degree polynomial through evenly spaced points on a smooth function and watching the fit oscillate hard between the points near the ends — not outside the data, between observations you actually have. Increasing the degree makes it worse rather than better, which is the opposite of the intuition that more flexibility means a better fit.

So "I only predict inside my range" does not buy the safety it sounds like it does. Inside the range still contains gaps between your observations, and gaps are where the wobble lives. If your x values are dense and even you will get away with more, but that is a property of your sampling, not of the method.

Worth naming the thing your eye is doing so you can stop trusting it here: scatter plots have arbitrary axis scales.

Perpendicular distance is not invariant to rescaling. Stretch the y axis and the principal axis rotates; the least squares line does not change at all in terms of which points it predicts well. So the fit that "looks right" depends on your plotting choices, and the fit that minimises prediction error does not.

That is a decent argument for the regression line being the more meaningful object of the two whenever prediction is the goal — it is a property of the data, not of the picture.

Worth adding which assumptions survive the switch, because it is not none of them.

Independence is not optional either way. If your rows are not independent — repeated measurements on the same customers, consecutive days of the same time series, students within the same schools — then your held-out set is not really held out. Nearby rows leak into each other and cross-validation reports an accuracy you will not see in production. That is a prediction problem, not an inference problem, and it bites hardest exactly where people feel safest.

Non-constant variance quietly changes what you are optimising. Squared error weights every observation equally, so if the noise is ten times larger in one region, that region dominates the fit. The model is not invalid, but it is spending its capacity where the signal is worst.

So the residual plot is still worth looking at in the machine learning setting. You are reading it for a different reason: not to check whether your p-values are legitimate, but to see where your model is structurally wrong.