48GB M4 Pro, Ollama, 14B Q4_K_M, 8k context. I get a steady 18 tok/s. Everything I read has people casually mentioning 40+ on Apple silicon and I cannot tell whether they are on different hardware, measuring something else, or whether I have left a setting somewhere.
Activity Monitor shows the GPU pinned, memory pressure green, nothing swapping. Is 18 just what this machine does?
18 is normal and here is how to check any claim you read.
Generation is memory-bandwidth bound. Every token requires reading essentially all the weights. An M4 Pro is around 273 GB/s; a 14B at Q4 is about 9GB of weights, so the theoretical ceiling is 273/9 which is roughly 30 tok/s, and real-world efficiency lands you at 60-70% of that. 18-21 is exactly where you should be.
The people posting 40+ are almost always on a Max or Ultra chip with 400-800 GB/s, or running a 7B, or quoting prompt processing rather than generation. Those are three different numbers and threads mix them constantly.
Divide bandwidth by model size. That is the first heuristic anyone has given me that lets me predict a number instead of comparing anecdotes.
It also tells you what a bigger quant costs before you download it. Going from Q4 to Q8 doubles the bytes read per token, so expect roughly half the tok/s. The maths holds up well enough for planning.
For interactive use 18 is fine - it is faster than you read. It only becomes a problem for batch work, and for batch you want a machine that can run several requests at once anyway, which is a different purchase.
Make sure you are comparing generation to generation. Prompt processing on Apple silicon is fast and gets quoted a lot, because 800 tok/s of prompt ingestion sounds much better in a post than 18 tok/s of output. Both are real numbers about the same run.
The first time I felt genuinely rich was buying a laptop with enough unified memory to hold a model, and the first time I felt poor again was reading the bandwidth spec.
@soy_pour_sasha · 9mo ago · 2 replies
Two things worth confirming anyway, both take a minute:
sudo sysctl iogpu.wired_limit_mbis the knob. Not your problem at 9GB, but remember it exists.Reply
Report
@epoxy_puddle · 9mo ago
Log says Metal, all layers offloaded. So it is just physics. Oddly relieving - I had spent an evening convinced I had installed something wrong.
Reply
Report