3060 12GB, and RAM is cheap enough that going to 96GB costs a fraction of what a bigger card would. I want to run 32b-class models and I can tolerate slow, but not two tokens a second slow. Everyone says VRAM is what matters, and yet the loaders will happily spill into system memory, so I want to understand what I am actually buying before I choose between more RAM and saving for a used 24GB card.
Politely against the buy-the-card consensus. RAM is so cheap relative to a 24GB card that I would buy it anyway, because it also buys you longer contexts, bigger batch jobs, and the ability to run something overnight without babysitting it. It is not either-or on any sane budget, it is RAM now and the card when you can. I ran a 70b overnight on mostly system memory to produce a dataset and I did not care that it took nine hours, because I was asleep.
The important exception is mixture-of-experts models, and it changes the answer materially. Because only a small fraction of the parameters are active per token, a large MoE can run at a perfectly pleasant speed with most of it sitting in system RAM. If your 96GB plan is aimed at a big MoE rather than a dense 32b, the RAM is a much better purchase than this thread would otherwise suggest. Dense models punish you for offloading; sparse ones mostly do not.
This is the correct nuance and it is why the blanket 'RAM is useless' answer has aged badly. Worth deciding which models you actually want to run before buying either thing.
Generation speed for a dense model is essentially a memory bandwidth problem, because every token has to read the weights. Your 3060 has roughly 360 GB/s. Dual channel DDR5 gives you somewhere around 90 to 100 GB/s in practice, so anything living in system RAM is being served at about a quarter of the card's rate, and that is before the transfer overhead. So more RAM does not make a 32b model fast, it makes it possible. The rule that predicts your speed is what fraction of the layers ended up off the card, not how much RAM you own.
And the gap is larger against the card you are saving for. A 3090 is comfortably over 900 GB/s, so a fully resident model on that card is roughly an order of magnitude better fed than anything running out of your DDR5.
That framing is what I was missing. I had been thinking of RAM as slower VRAM rather than as a different order of magnitude.
Do not forget the KV cache. At long context it takes a serious bite out of your 12GB, which pushes more layers off the card and makes the whole thing worse exactly when you are asking the most of it. Test at the context length you will really use, not at 2k.
Cheapest experiment available: you can already test this. Run the 32b quant you want with your current RAM and as much offload as fits, and measure. If it comes out at three tokens a second, more RAM will let you load bigger files at three tokens a second. RAM changes what fits, not what is fast.
@yamlfatigue · 3mo ago
Numbers from my own machine, which is close to yours: 12GB card, 64GB of DDR5. A 32b at a four-bit quant with about half the layers offloaded gives me something in the three to four tokens a second range, and it degrades further as the context fills because the KV cache competes for the same VRAM. It is usable for a batch job you walk away from and genuinely unpleasant to chat with. Below about eight tokens a second I stop using a model interactively, and everybody I know has a threshold like that.
Reply
Report