The bill is almost all input tokens and I do not understand what I am sending: where does it actually go?

Measure before cutting, because the intuition is usually wrong about which of those four is your problem.

Log input and output tokens separately on every call, along with a label for what kind of call it was. Most SDKs return the counts in the response and people throw them away.

Then for a representative sample, break the input down:

• Tokens in the system prompt, fixed, easy to count once
• Tokens in tool definitions, same
• Tokens in retrieved context, varies per call, worth an average
• Tokens in history, the remainder, and usually the answer

Half an hour and you will know which lever matters. I have seen all four be the dominant one in different products, which is why general advice is useless here.

The metric worth watching afterwards is input tokens per user action, not total spend. Total spend moves with traffic and tells you nothing about whether a change helped; per action isolates the thing you control.

And log it per feature. Usually one flow accounts for most of the bill, and it is rarely the one you would guess.

Input dominating is the normal shape and the reason is structural: you send everything every time.

A model has no memory between calls. So on turn ten of a conversation you are not sending turn ten, you are sending the system prompt, plus all nine previous turns, plus the new message. The cost of a conversation grows with the square of its length, and nothing in the interface makes that visible.

Where the tokens usually are, in order:

Conversation history. Dominant in anything multi-turn. Long conversations are where budgets disappear.

Retrieved context. If you inject documents, this is often enormous relative to the question. Five chunks of a thousand tokens is five thousand tokens to answer a ten token question.

The system prompt. Fixed per call, so it matters in proportion to how many calls you make. A two thousand token system prompt on a million calls is a real number.

Tool definitions. Frequently overlooked. Every tool schema is sent on every call, and a dozen verbose schemas is thousands of tokens per request, whether or not any tool is used.

That last one catches people badly, because it is invisible in the code: you define the tools once and pay for them forever.

Then the levers, in rough order of return per effort:

Prompt caching. If your provider offers it, the stable prefix - system prompt, tool definitions, anything constant: can be cached and charged at a large discount on subsequent calls. This is the single highest-return change for most products and it requires no reduction in what you send.

The catch is that it works on a prefix, so ordering matters enormously: everything stable must come first and everything variable last. A system prompt that interpolates the current date at the top invalidates the cache on every call, and people do exactly that.

Trim the history. Send a rolling window rather than the whole conversation, or summarise older turns into a short block. Most products do not need turn one verbatim at turn forty.

Retrieve less, better. Five mediocre chunks cost more and answer worse than two good ones. A reranker that lets you send three instead of ten usually pays for itself.

Prune tool definitions. Send only the tools relevant to the current step rather than all of them.

Shorten the system prompt last. It is the most visible and usually the smallest share, which is why people start there and see nothing change.