hosted api or a $290/mo gpu box for 2.4m tokens a day AI Spend

You are missing utilisation, which is the whole argument.

2.4M tokens a day spread evenly is about 28 tokens/second sustained. A 24GB card running a quantised 7-8B model with proper batching will do many multiples of that. So you would be renting a machine that sits idle most of the time, and you pay for idle at exactly the same rate as work.

The hosted API charges you for tokens. The box charges you for time. Self-hosting wins when you can keep the box busy - which means either much higher volume, or bursty work you can queue up and run flat out for four hours a night rather than trickling all day.

Your workload is actually a good candidate for that second shape, because you said nothing is real time. Batch it. If you can process a day's documents in a 90 minute window at high throughput, the economics change completely, and you could even use a cheaper spot instance because you do not care about interruptions.

Costs you did not list on the self-hosted side:

• your time. Model updates, driver issues, the OOM at 3am, the retry logic for a box that is not five nines. Call it 3-4 hours a month at a minimum, more in the first two.
• redundancy. One box is one failure away from your pipeline stopping. The hosted API's uptime is somebody else's problem and that is worth something you should price honestly.
• the evaluation work to prove the open model is actually good enough at your task. This is real and people skip it.

At a $50/month delta, the hosted API is cheaper once you value your own time above zero. Come back to this at 20M tokens a day, where the delta is large enough to pay for the operational burden.

Before choosing where to run it, cut what you send. On a classification workload people routinely ship the entire document when the first and last 800 tokens decide the label 95% of the time.

We cut input tokens by 62% with a truncation heuristic and lost half a point of accuracy. That is a bigger cost reduction than any infrastructure decision on the table here, and it applies whichever option you choose.

Also, if your prompts share a long fixed prefix - instructions, label definitions, few-shot examples - prompt caching on the hosted side makes the input portion much cheaper. Your workload is 80% input, so that is aimed directly at your biggest line item.

Practical numbers from running this, so you can sanity check whatever you pick.

On a 24GB card, a 7-8B model at Q4_K_M leaves comfortable room for a reasonable context window. Single-stream generation is somewhere around 45-60 tok/s depending on card and quantisation. That number is the one everyone quotes and it is misleading, because with a proper serving stack and continuous batching you get many times that in aggregate throughput. If you benchmark with one request at a time and conclude self-hosting is slow, you have benchmarked the wrong thing.

Quantisation notes for a classification workload specifically: Q4 is generally fine, Q8 barely differs from full precision for this kind of task, and your accuracy is far more sensitive to prompt structure than to the last bit of precision. Build an eval set of 200 labelled documents before you touch any of this and score every option against it. Otherwise you are choosing infrastructure based on how the output feels, which is how people end up paying more for worse results.

Context window is the thing that will actually bite you on 24GB. KV cache grows with context and concurrency, so long documents plus batching is what pushes you into out-of-memory, not the weights.

Hybrid is worth considering and rarely gets mentioned. Run the small open model locally for the easy 80%, and escalate the low-confidence cases to the hosted API. You get most of the cost saving with a quality floor set by the better model.

It does mean maintaining two paths and a confidence threshold you have to tune, so it is not free. But for high-volume classification it is often the right shape once you are past the point where one option is obviously cheaper.

Past about a million tokens a day self-hosting is always cheaper, that is roughly the crossover everyone finds. At 2.4M you are well past it, buy the box.

$50/month of savings in exchange for becoming a person who owns a GPU that can page you.