Small internal tool, maybe 40 requests a day in bursts. If two people use it within a few minutes it is instant. If nobody has touched it for an hour, the first request takes 38 seconds before a single token appears,…
At 4k context I get 28 tok/s and everything is on the card. At 16k the same model gives me 3.1 tok/s. No error, no warning, it just gets slow. shows 23.9GB used, and the server log says it offloaded 51 of 65 layers to…
4060 Ti 16GB. The task is document extraction: about 3k tokens of prompt in, a JSON object with 12 fields out, several of them nullable. Runs unattended on a few hundred documents a night, so correctness matters more…
8B model, Q4 K M, so everything should be on the GPU. says 6.9GB on the card, which matches what I expect. But the process RSS is 5.4GB. On a 16GB box running two other things, that is the difference between fine and…
$700 total. Motherboard has one PCIe x16 slot and one x4, 750W PSU, and I would rather not replace either. Two 3060 12GB cards gets me 24GB for around $500 with room left over. One used 3090 is 24GB for roughly $650 700…
48GB M4 Pro, Ollama, 14B Q4 K M, 8k context. I get a steady 18 tok/s. Everything I read has people casually mentioning 40+ on Apple silicon and I cannot tell whether they are on different hardware, measuring something…