If the likelihood of my data is 10 to the minus 300, in what sense did the correct model produce it?
I simulated data from a normal distribution and then computed the likelihood of that data under the very distribution that generated it. The number came out astronomically small.
That is what confuses me. This is the correct model — the one I know is right, because I used it to make the data. If the right answer scores essentially zero, what is the score measuring, and how can a method built on maximising it be sound?
I suspect I am misreading what the number represents but I cannot see how.
@residual_ruth · 3w ago
The clarifying experiment is two minutes and it makes the point better than an explanation.
Compute the likelihood of your data under the true parameters, and then under parameters that are noticeably wrong — mean shifted by two, say. Both numbers will be unreadably small. Now take the ratio, or equivalently the difference of the log-likelihoods.
You will find the true parameters beat the wrong ones by an enormous factor. That factor is the entire content of the method. Maximum likelihood never asks "is this number big" — it asks "which parameter values make this number biggest", and the answer to that is perfectly stable regardless of how small everything is in absolute terms.
Reply
Report