32b q4_k_m drops to 3 tok/s at 16k context on a 24gb 3090 vram-fit

The KV cache is what pushed you over. For a 32B with GQA it is roughly 0.25 MB per token at fp16, so 4k context is about 1 GB and 16k is about 4 GB. Your weights are 19.9 GB and never move; the cache is the variable, and 19.9 + 4 + about 1 GB of compute buffers does not fit in 24.

Cheapest fix, in order:

1. Quantise the KV cache. Q8 halves it with essentially no quality change on a 16k window. In llama.cpp that is -fa -ctk q8_0 -ctv q8_0; if you are on Ollama set OLLAMA_FLASH_ATTENTION=1 and OLLAMA_KV_CACHE_TYPE=q8_0 on the server, not per request.
2. Get the display off the card. A desktop session holds 0.5-1 GB, which at this margin is decisive.
3. Ask whether you need 16k. Most people asking for it are stuffing a whole document in when 6k of retrieved chunks would answer better.

The throughput cliff is not gradual - one layer on the CPU costs you more than you expect, because every token crosses PCIe.

Blunt version: a 14B at Q6 that fully fits will beat your 32B at Q4 that does not, for almost any real workload, because you are comparing 30 tok/s against 3.

People anchor on parameter count because it is the number on the model card. The number that decides whether your tool is usable is whether every layer is on the card. Below that line nothing else matters.

The line to watch in the logs is the layer count - offloaded 51/65 layers to GPU. If that is not 65/65 you are not measuring what you think you are measuring, and every benchmark you post is meaningless.

I keep it in my shell prompt for the server, effectively. Saved me from a lot of confused comparisons.

Set num_gpu higher and force all 65 layers on. The runtime is being conservative with the estimate, it will fit if you tell it to.

If it is a headless box, check nothing else grabbed VRAM. I lost 900MB for a month to a container that had a CUDA context open and was doing nothing at all with it.