llama.cpp with -ngl 99 still eats 5gb of system ram, where is it going llama.cpp

Most of it is not really "used". The model file is memory-mapped, so the pages you have read count towards RSS but they are page cache backed by the file on disk - the kernel will drop them the instant something else needs the memory. A 4.7GB GGUF that has been fully read shows up as roughly 4.7GB of RSS and costs you almost nothing.

Prove it to yourself: run with --no-mmap and watch RSS become genuinely smaller after load, while startup gets slower. Or look at PSS instead of RSS - smem -k -P llama will show you a much more honest number.

If it is actually swapping, that is a separate problem and it is usually the other two things on the box, not this.

The genuinely host-resident parts, so you know what is left after mmap:

• the compute buffer for the largest batch, which scales with batch size and context
• the KV cache, but only if you passed --no-kv-offload, otherwise it is on the card
• vocab and tokenizer structures, small
• whatever your OS decided to keep of the file

On an 8B with default settings the real host cost is a few hundred MB. If you measure more than that after --no-mmap, something else is going on.

Related and worth knowing before it bites: with mmap the first generation after a cold start is slow because the pages get faulted in as they are touched. If you are benchmarking and your first run is always terrible and every later run is fine, that is this, not the model warming up in any meaningful sense.

"I can hear it" is the most reliable performance monitoring tool I own and I am not being sarcastic.

Since the real complaint is swapping: set the container memory limits, and if you are on Linux with fast storage consider zram over a swapfile. Any inference workload that touches swap during generation goes from tolerable to unusable within one token.