Ask
18
@prior_priya ·

If the likelihood of my data is 10 to the minus 300, in what sense did the correct model produce it?

I simulated data from a normal distribution and then computed the likelihood of that data under the very distribution that generated it. The number came out astronomically small.

That is what confuses me. This is the correct model — the one I know is right, because I used it to make the data. If the right answer scores essentially zero, what is the score measuring, and how can a method built on maximising it be sound?

I suspect I am misreading what the number represents but I cannot see how.

5 answers Share
Report

Answering anonymously — a moderator will review it first.

  • @residual_ruth · 3w ago

    The clarifying experiment is two minutes and it makes the point better than an explanation.

    Compute the likelihood of your data under the true parameters, and then under parameters that are noticeably wrong — mean shifted by two, say. Both numbers will be unreadably small. Now take the ratio, or equivalently the difference of the log-likelihoods.

    You will find the true parameters beat the wrong ones by an enormous factor. That factor is the entire content of the method. Maximum likelihood never asks "is this number big" — it asks "which parameter values make this number biggest", and the answer to that is perfectly stable regardless of how small everything is in absolute terms.

    21
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
  • @logit_lorenzo · 3w ago

    You are reading it as an absolute score when it is only ever a relative one. Two things are making it tiny, and neither says anything is wrong.

    It is a product over every observation. Each data point contributes a factor less than 1, so with 500 points you are multiplying 500 small numbers. 10^-300 for a few hundred observations is completely ordinary — the exponent scales with sample size, so the number carries no information about fit on its own.

    For continuous data it is a density, not a probability. The probability of drawing any exact real value is zero. What you computed is a density height, which is only meaningful multiplied by an interval width. It can even exceed 1 for a sufficiently narrow distribution, which is a good sign that it is not a probability.

    So the correct answer to "how can 10^-300 be the right model" is that the wrong models score 10^-340, and you would only ever have looked at the ratio.

    25
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
  • @bootstrap_bo · 3w ago

    Practical note that follows from all this: never compare log-likelihoods across datasets of different sizes, and be careful comparing across different transformations of the outcome.

    More observations means a more negative log-likelihood automatically, so "model A on last year's data scored -4200 and model B on this year's scored -5100" is not a comparison of anything. The information criteria people quote — AIC and friends — exist partly to make comparisons legitimate within one dataset, and they still do not license comparisons across datasets.

    If you take one habit from the thread: a lone likelihood number is not evidence. A difference between two likelihoods on the same data is.

    15
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
  • @sample_sena · 3w ago · 2 replies

    Also worth knowing why nobody sees this in practice — every library returns the log, and reports differences. You ran into it because you computed the raw product yourself, which is exactly the thing the tooling is arranged to avoid.

    9
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
    • @prior_priya · 3w ago

      "The wrong models score 10^-340" is the sentence that fixed it. I was treating the likelihood as a goodness score on some absolute scale, and it never was one.

      7
      Share
      Reply

      Answering anonymously — a moderator will review it first.

      Report