Ask
26
@logit_lorenzo ·

Why did nobody in my machine learning course mention the regression assumptions my statistics course insisted on?

I learned linear regression twice. In the statistics course it came with a list of conditions to check — residuals roughly normal, constant variance, independence, no strong collinearity — and a warning that the model is not trustworthy otherwise.

Then I learned it again from a machine learning angle, and none of that appeared. It was a model with parameters, fit it by minimising squared error, evaluate on held-out data, done. Residual plots did not come up once.

One of these framings is leaving something out and I would like to know which. Are the assumptions quietly still required and the machine learning course was sloppy, or do they genuinely not apply when you use the model differently?

5 answers Share
Report

Answering anonymously — a moderator will review it first.

  • @residual_ruth · 2w ago

    Neither course is sloppy. They are answering different questions, and the assumptions belong to one of the questions and not the other.

    The statistics framing is aimed at inference: is this coefficient different from zero, how wide is the interval around it, can I say this predictor matters. Every one of those outputs is a probability statement, and a probability statement needs a probability model. The normality and constant-variance conditions are what license the standard errors, the p-values and the confidence intervals. Break them and those numbers are still printed, but they are wrong.

    The machine learning framing is aimed at prediction: how close is the predicted value to the real one on data the model has not seen. That question has an empirical answer — hold data out and measure. You are not making a probability claim about a coefficient, so you do not need the machinery that would justify one.

    So the honest version is: the assumptions are conditions for the inferential outputs, not for the fitting. Drop the outputs and you drop the conditions with them.

    28
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
  • @prior_priya · 2w ago

    One more that catches people moving in the other direction — from machine learning to statistics, on the same data.

    Collinearity is close to harmless for prediction and fatal for interpretation. Two nearly identical predictors will split the credit between them in an essentially arbitrary way; predictions barely move, individual coefficients swing wildly and can flip sign between refits.

    This becomes a problem the moment somebody looks at your predictive model and asks "so which factor matters most?" — a question that quietly moves you from the framing where the assumptions did not matter to the one where they do. That transition usually happens in a meeting rather than in the code, which is why nobody notices it.

    17
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
  • @overfit_omar · 2w ago

    Worth adding which assumptions survive the switch, because it is not none of them.

    Independence is not optional either way. If your rows are not independent — repeated measurements on the same customers, consecutive days of the same time series, students within the same schools — then your held-out set is not really held out. Nearby rows leak into each other and cross-validation reports an accuracy you will not see in production. That is a prediction problem, not an inference problem, and it bites hardest exactly where people feel safest.

    Non-constant variance quietly changes what you are optimising. Squared error weights every observation equally, so if the noise is ten times larger in one region, that region dominates the fit. The model is not invalid, but it is spending its capacity where the signal is worst.

    So the residual plot is still worth looking at in the machine learning setting. You are reading it for a different reason: not to check whether your p-values are legitimate, but to see where your model is structurally wrong.

    22
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
  • @sample_sena · 2w ago · 2 replies

    The clean way I have heard it put: statistics asks what the world is like, machine learning asks what happens next. Assumptions are how you make a claim about the world with a straight face; held-out data is how you check what happens next.

    Both courses were teaching linear regression. Only one of them was teaching it as a claim.

    13
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
    • @logit_lorenzo · 2w ago

      That reframing plus "independence is not optional either way" is what I was missing. I had assumed the machine learning course dropped all four, when actually it dropped the two that only support inference.

      10
      Share
      Reply

      Answering anonymously — a moderator will review it first.

      Report