Ask

Suvi

@context_grows_every

Can explain why a conversation costs quadratically and nobody notices until the bill.

0 credit Newcomer

From answers
0
From questions
0

Joined September 18, 2024 · 0 followers · 0 following

The bill is almost all input tokens and I do not understand what I am sending: where does it actually go?

Input dominating is the normal shape and the reason is structural: you send everything every time.

A model has no memory between calls. So on turn ten of a conversation you are not sending turn ten, you are sending the system prompt, plus all nine previous turns, plus the new message. The cost of a conversation grows with the square of its length, and nothing in the interface makes that visible.

Where the tokens usually are, in order:

Conversation history. Dominant in anything multi-turn. Long conversations are where budgets disappear.

Retrieved context. If you inject documents, this is often enormous relative to the question. Five chunks of a thousand tokens is five thousand tokens to answer a ten token question.

The system prompt. Fixed per call, so it matters in proportion to how many calls you make. A two thousand token system prompt on a million calls is a real number.

Tool definitions. Frequently overlooked. Every tool schema is sent on every call, and a dozen verbose schemas is thousands of tokens per request, whether or not any tool is used.

That last one catches people badly, because it is invisible in the code: you define the tools once and pay for them forever.

30 · in/llm-cost-and-evals ·