How much GPU memory does a 7B model actually need, and why do the numbers people quote vary so much?

Training is a completely different budget, which is the other half of why quoted figures vary so wildly. For full fine-tuning you need, roughly:

• the weights, plus
• the gradients, about the same size again, plus
• the optimiser state, which for the common optimiser is roughly twice the weights again, plus
• activations, which depend on batch size and sequence length.

That lands full fine-tuning of a 7B model somewhere around 60–80 GB — well beyond a single consumer card, which is why nobody does it that way at home.

What people actually do is parameter-efficient fine-tuning: freeze the weights and train a small number of added parameters. Then the gradient and optimiser terms shrink to almost nothing and the job fits alongside a quantised model on a single consumer card. If you see somebody fine-tuning a 7B model on one GPU, this is what they are doing.

There is arithmetic and it is simple enough to do in your head once you know the terms.

Start with the weights. Memory for weights is roughly parameters × bytes per parameter. So for seven billion:

• 16-bit (half precision): 2 bytes each → about 14 GB
• 8-bit quantised: 1 byte → about 7 GB
• 4-bit quantised: half a byte → about 3.5 GB

That is where the whole range in your question comes from, and it is the single biggest factor.

Then add overhead for inference: the runtime itself, and the key-value cache that grows with how much context you are holding. Budget roughly 1–3 GB on top for a modest context, more if you want a long one — the cache can become the dominant term at very long contexts.

So a 4-bit 7B model fits comfortably on an 8 GB card; a 16-bit one needs 16 GB and is tight.

Two practical points the arithmetic does not capture.

Quantisation costs quality, and how much depends on the level. Going to 8-bit is close to free for most purposes. 4-bit is usually acceptable and noticeably worse on some tasks. Below that it degrades quickly. So the memory saving is real and it is not free, and the right level is the one where your own evaluation still passes.

You do not need a GPU at all for a 7B model. Running on the processor with system memory works, and it is slower — usable for occasional queries, painful for anything interactive. On machines with unified memory, the split between the two is less meaningful and the practical limit is total memory.

Worth trying on what you already own before buying anything. The arithmetic tells you whether it fits; only running it tells you whether the speed is tolerable for what you want.

One more term that surprises people when they first hit it: context length is not free at inference.

The cache holding attention state grows with the number of tokens you are keeping, and at long contexts it can exceed the size of the weights. So a model that loads comfortably can run out of memory partway through a long conversation, which reads like a leak and is not.

If you plan to use long contexts, budget for it explicitly rather than sizing on the weights alone.