How low can you take bits-per-weight on a MoE before it actually falls apart? (and what does bpw mean exactly)

What breaks first is the thing you will not notice in a ten-minute chat.

Short conversational quality is remarkably robust to aggressive quantisation. What degrades early is long-horizon behaviour: holding an output format across many turns, following a multi-step instruction to the end, using tools correctly on the fortieth call rather than the fourth, not quietly dropping a constraint it agreed to twenty messages ago.

So the standard failure story is: person tries a low quant, it feels great, they make it the daily driver, and a week later they cannot work out why their agent loop has become unreliable. Nothing crashed. It just got slightly worse at staying on task, which is the hardest kind of regression to attribute.

Bits per weight, yes — the average number of bits used to store each parameter. That is why it comes out as an odd fraction rather than a round number.

The reason it is an average is that quantisation is not applied uniformly. Different tensors get different treatment, and on a mixture-of-experts model that gap is wide: the router and the shared expert are often kept at high precision because errors there are catastrophic — a wrong routing decision does not degrade the answer, it sends you to the wrong expert entirely — while the routed expert weights get pushed much lower because there are so many of them and they carry most of the file size.

So a file described as 4.8bpw might be, very roughly, 8-bit routing and embeddings plus 4-bit experts, averaged over a parameter count dominated by the experts. Two files with the same average can have quite different layouts and quite different behaviour.

Do not use perplexity as your proxy. It is easy to compute and it moves smoothly, which makes it feel scientific, but it is dominated by high-frequency tokens and can barely move while the behaviour you care about degrades badly.

Use task evaluations against work you actually do. Twenty items, pass/fail, scored the same way every time. It is less elegant and much more predictive.

Remember that KV cache quantisation is a separate knob from weight quantisation, and on a memory-constrained box it is often the better trade.

Quantising the cache costs you some quality over very long contexts but leaves the weights alone. If your problem is "the model I want does not leave room for the context I need", try that before you go down a bit on the weights — you frequently get the headroom you were after without touching model quality at all.

Honestly below 8 bits it is all noise anyway — the weights are mostly redundant, so quantisation is effectively lossless until you get very low. That is why nobody can tell the difference in blind tests.