Small internal tool, maybe 40 requests a day in bursts. If two people use it within a few minutes it is instant. If nobody has touched it for an hour, the first request takes 38 seconds before a single token appears, then normal speed after that.
20GB model on a SATA SSD. I assume it is being unloaded and read back off disk, but I would like to confirm before I buy an NVMe drive that might not fix anything.
It is being unloaded - the default keep-alive is 5 minutes and yours matches exactly. Do not buy a drive, change a setting.
Per request: "keep_alive": -1 in the JSON body keeps it resident indefinitely.
Server-wide, which is what you want for a shared tool: set OLLAMA_KEEP_ALIVE=-1 in the environment of the server process. If it runs under systemd that means a drop-in with Environment="OLLAMA_KEEP_ALIVE=-1" and a daemon-reload, not an export in your shell - that trips up everyone at least once.
Then preload on boot by sending one request with an empty prompt so the first real user is never the one who pays.
systemd drop-in did it. First request after eleven hours idle was 900ms. I was about to spend €90 on a drive.
Just be aware you are now pinning ~20GB of VRAM permanently. Fine on a dedicated box, less fine if anyone else uses that GPU for anything, because their allocation will fail rather than queue.
40 requests a day in bursts is exactly the profile where a small always-resident model beats a big one that has to be paged in. Worth checking whether the 20GB is buying you anything for an internal tool.
If for some reason you cannot set the environment (managed box, someone else's service file), a cron that pings the model every 4 minutes with a one-token generation achieves the same thing for the cost of a few hundred wasted tokens a day. Ugly, works.
Watch out for the related one: if you serve two models and the second gets requested, the first can be evicted to make room and now you have two reload cliffs alternating. OLLAMA_MAX_LOADED_MODELS controls how many stay resident, but only if the VRAM is actually there for both.
@two_calendars · 8h ago · 3 replies
Your 38 seconds is arithmetic, which is a nice confirmation: 20GB off a SATA SSD at about 530MB/s is 37-38s of pure read. That is exactly your number, so nothing else is wrong.
An NVMe at 3.5GB/s would make it about 6 seconds, which is a real improvement if you ever do need to reload - but keeping it resident makes it zero, so fix that first and buy the drive only if you end up juggling models.
Reply
Report
@bedtime_bargain · 3h ago
The fact that the number matched the drive spec so exactly is the most satisfying debugging moment I have had this year.
Reply
Report
@teardown_tues · 3h ago
Worth doing this arithmetic any time a load feels slow. If your measured time is well above file size divided by drive speed, the disk is not the bottleneck and you are about to buy the wrong hardware.
Reply
Report