Ask
22
@bootstrap_bo ·

My least squares line looks visibly tilted compared to the cloud of points — is the fit wrong?

I generated correlated test data, fitted an ordinary least squares line and plotted both. The line does not look like it goes through the middle of the cloud. It looks rotated, sitting shallower than the long axis of the scatter.

Out of curiosity I computed the principal axis of the data instead, and that line looks like what I expected the fit to be — it goes straight down the spine of the cloud.

So either the regression is doing something I have misunderstood, or my eyes are wrong about what a good fit looks like. Which is it, and if the regression is right, what is my eye responding to?

5 answers Share
Report

Answering anonymously — a moderator will review it first.

  • @residual_ruth · last wk.

    Your eyes are fine and so is the regression. They are measuring different distances.

    Ordinary least squares minimises the vertical distance from each point to the line — the error in y only, with x treated as known. Your eye, and the principal axis, minimise perpendicular distance, which treats the two directions symmetrically.

    Those give different lines whenever the data is noisy, and the regression line is always the shallower of the two. That is not a defect; it follows from the question being asked. "Given this x, what is my best guess for y" is not symmetric in x and y, so its answer should not be either.

    The visual check that makes this click: draw the vertical segment from each point to the line rather than the perpendicular one. Under that view the regression line is obviously centred and the principal axis obviously is not.

    29
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
  • @logit_lorenzo · last wk.

    The consequence worth knowing is that the two lines answer different practical questions, so which one you want depends on what you are going to do with it.

    • Predicting y from x — measurement, forecasting, any "given this input, estimate this output" task. Ordinary least squares, vertical distances, and the shallower line is genuinely the better predictor. Its shallowness is regression to the mean doing its job.
    • Estimating an underlying relationship where both variables are measured with error — two instruments measuring the same thing, calibration between methods. Then treating x as known is a lie, and the perpendicular fit is closer to what you mean.

    A useful sanity check for the first case: regress y on x, then regress x on y, and plot both. You get two different lines, and neither is the principal axis. If those two lines are far apart your data is noisy enough that the distinction matters; if they nearly coincide it does not.

    24
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
  • @overfit_omar · last wk.

    Worth naming the thing your eye is doing so you can stop trusting it here: scatter plots have arbitrary axis scales.

    Perpendicular distance is not invariant to rescaling. Stretch the y axis and the principal axis rotates; the least squares line does not change at all in terms of which points it predicts well. So the fit that "looks right" depends on your plotting choices, and the fit that minimises prediction error does not.

    That is a decent argument for the regression line being the more meaningful object of the two whenever prediction is the goal — it is a property of the data, not of the picture.

    16
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
  • @prior_priya · last wk. · 2 replies

    Small addition since it surprises people: the regression line does pass through the point (mean of x, mean of y), always. So it is genuinely centred in the cloud, just at a shallower angle than the cloud's spine.

    Plot that point and it stops looking like the line has been pushed off to one side.

    10
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
    • @bootstrap_bo · last wk.

      Drawing the vertical residual segments was the fix. Under that view the shallow line is obviously the balanced one and the principal axis leaves systematic error at both ends.

      8
      Share
      Reply

      Answering anonymously — a moderator will review it first.

      Report