Why did nobody in my machine learning course mention the regression assumptions my statistics course insisted on?
I learned linear regression twice. In the statistics course it came with a list of conditions to check — residuals roughly normal, constant variance, independence, no strong collinearity — and a warning that the model is not trustworthy otherwise.
Then I learned it again from a machine learning angle, and none of that appeared. It was a model with parameters, fit it by minimising squared error, evaluate on held-out data, done. Residual plots did not come up once.
One of these framings is leaving something out and I would like to know which. Are the assumptions quietly still required and the machine learning course was sloppy, or do they genuinely not apply when you use the model differently?
@residual_ruth · 2w ago
Neither course is sloppy. They are answering different questions, and the assumptions belong to one of the questions and not the other.
The statistics framing is aimed at inference: is this coefficient different from zero, how wide is the interval around it, can I say this predictor matters. Every one of those outputs is a probability statement, and a probability statement needs a probability model. The normality and constant-variance conditions are what license the standard errors, the p-values and the confidence intervals. Break them and those numbers are still printed, but they are wrong.
The machine learning framing is aimed at prediction: how close is the predicted value to the real one on data the model has not seen. That question has an empirical answer — hold data out and measure. You are not making a probability claim about a coefficient, so you do not need the machinery that would justify one.
So the honest version is: the assumptions are conditions for the inferential outputs, not for the fitting. Drop the outputs and you drop the conditions with them.
Reply
Report