Why does everything switch to log probabilities the moment the maths gets serious?

Range first, because it is short: log of 1 is 0, and log of anything smaller is negative, running down towards minus infinity as the probability approaches 0. So log probabilities live in (-infinity, 0], and closer to zero is more likely.

That is worth saying out loud because it flips the direction people expect. -2 is a much better score than -8420. When you see a log-likelihood of -8420 your reaction should not be "that is a huge negative number, something is broken" — it should be "that is a number I can only interpret by comparing it to another one from the same data."

Which is the real habit to build: log probabilities are almost never meaningful alone. They are meaningful as differences. A difference of 2 in log space means one thing is about 7.4 times more likely than the other, and that ratio is the thing you actually care about.

For question 2 — the numerical argument is real and it is not marginal.

Multiply a thousand probabilities of 0.5 together and you have 0.5^1000, which is around 10^-301. That is close enough to the floor of double precision that a few thousand more observations underflow to exactly 0. And once it is 0, it is not "very small", it is gone: you cannot divide by it, you cannot compare two of them, and every ratio you were about to compute is 0/0.

In logs the same quantity is a sum of a thousand copies of about -0.693, which is around -693. Perfectly ordinary number, no drama.

So yes, mostly numerical — but the reason it bites so fast is worth internalising. Likelihood of a dataset is a product over observations, so the exponent grows linearly with your sample size. Underflow is not an edge case you might hit; with any real dataset it is the default outcome.

There is an answer to question 2 that survives perfect arithmetic, and it is the one I would actually lead with.

Optimisation and calculus both prefer sums. Almost everything we do to a likelihood involves differentiating it, and the derivative of a product of a thousand terms is a nightmare of product rules, while the derivative of a sum of a thousand terms is a sum of a thousand derivatives. Each term depends on its own observation and nothing else, which is exactly what makes the gradient cheap and what makes it parallelise.

The same trick shows up wherever products of many things appear — path probabilities, sequence models, hidden Markov chains. Logs are not a numerical patch bolted on; they turn the problem into the shape that the tools were built for.

And for the exponential family — normals, Poissons, gammas — the log cancels the exp that is sitting in the density anyway, and you are often left with something polynomial. That is not a coincidence, it is why those distributions are convenient in the first place.

"Closer to zero is more likely, and only differences mean anything" is the sentence I needed. I had been reading large negative numbers as failures.

The log-sum-exp warning is timely too — I have exactly that expression in a script and it has only ever run on toy data.

Question 3, what you lose: adding is now the hard operation.

Multiplying is easy in log space — you add. But if you need the probability of A or B, you need to add the probabilities, and there is no clean way to add in logs. You end up with the log-sum-exp trick: pull out the largest term, exponentiate the differences (all of which are now safely small), sum, take the log, add the largest term back.

Every library has this built in and you should use theirs rather than writing it, because the whole point is the reordering that keeps the exponentials in range. Writing log(sum(exp(x))) literally reintroduces the exact overflow you switched to logs to avoid, and it will work fine on your test data and fail on the real thing.