Why did nobody in my machine learning course mention the regression assumptions my statistics course insisted on?
I learned linear regression twice. In the statistics course it came with a list of conditions to check: residuals roughly normal, constant variance, independence, no strong collinearity - and a warning that the model is not trustworthy otherwise.
Then I learned it again from a machine learning angle, and none of that appeared. It was a model with parameters, fit it by minimising squared error, evaluate on held-out data, done. Residual plots did not come up once.
One of these framings is leaving something out and I would like to know which. Are the assumptions quietly still required and the machine learning course was sloppy, or do they genuinely not apply when you use the model differently?
@residual_ruth · last mo.
Neither course is sloppy. They are answering different questions, and the assumptions belong to one of the questions and not the other.
The statistics framing is aimed at inference: is this coefficient different from zero, how wide is the interval around it, can I say this predictor matters. Every one of those outputs is a probability statement, and a probability statement needs a probability model. The normality and constant-variance conditions are what license the standard errors, the p-values and the confidence intervals. Break them and those numbers are still printed, but they are wrong.
The machine learning framing is aimed at prediction: how close is the predicted value to the real one on data the model has not seen. That question has an empirical answer, hold data out and measure. You are not making a probability claim about a coefficient, so you do not need the machinery that would justify one.
So the honest version is: the assumptions are conditions for the inferential outputs, not for the fitting. Drop the outputs and you drop the conditions with them.
Reply
Report