Ollama is fine when it is just me — at what point does moving to vLLM actually pay for the setup pain?

Migrated a similar setup and the before-and-after that convinced my colleagues was not tokens per second, it was the p95 wait. Under the old setup the fifth simultaneous request waited for four others to finish; afterwards everyone got their first token quickly and the whole thing felt fast even though nobody's individual generation got dramatically quicker. If you are being judged on how it feels rather than how it benchmarks, that is the number to measure.

Eight concurrent users on one card is exactly the case vLLM is built for, so yes. The mechanism is continuous batching: instead of finishing one request and starting the next, it keeps a running batch and slots new sequences in as others finish, so the GPU stays busy. Single-user speed will be roughly what you have now, maybe a little better; aggregate throughput under concurrency is several times higher in my testing, and more importantly the tail latency stops being awful for whoever arrives last. The cost is that it is a server, not an app — you pin a model, you configure memory, and it is much less forgiving than Ollama about changing your mind.

Disagreeing with the migrate-now consensus. The llama.cpp server does continuous batching with parallel slots too, and for eight users on a 14b it may well be enough — you can enable multiple slots and set a sensible context split, and you keep the format you already have. Try that on an afternoon before you spend a weekend. If it holds up, you have solved the problem without changing your stack; if it does not, you have lost an afternoon and you now know the shape of your load.

Whatever you pick, put a queue and a per-user rate limit in front of it before you tell eight people it exists. The failure mode is not the engine, it is one colleague pasting a 200 page PDF into it on Monday morning and everybody else concluding the tool is broken.

The practical trap on a single 24GB card is memory. vLLM preallocates a large share of VRAM for the KV cache on startup based on a utilisation setting, which is what gives you the batching, and it will happily refuse to start if the weights plus that reservation do not fit. It also does not do the casual CPU offload you may be used to, and quantised formats are not interchangeable — the GGUF you are running today is not the artefact you will be serving. Budget your weekend for finding a suitable quantised copy of the model and getting max sequence length and memory utilisation to agree with each other, not for the install.

Eight internal users, one card, one model. That is a server workload, so use the server. The weekend is real and you do it once.