Measure before cutting, because the intuition is usually wrong about which of those four is your problem.
Log input and output tokens separately on every call, along with a label for what kind of call it was. Most SDKs return the counts in the response and people throw them away.
Then for a representative sample, break the input down:
- Tokens in the system prompt, fixed, easy to count once
- Tokens in tool definitions, same
- Tokens in retrieved context, varies per call, worth an average
- Tokens in history, the remainder, and usually the answer
Half an hour and you will know which lever matters. I have seen all four be the dominant one in different products, which is why general advice is useless here.
The metric worth watching afterwards is input tokens per user action, not total spend. Total spend moves with traffic and tells you nothing about whether a change helped; per action isolates the thing you control.
And log it per feature. Usually one flow accounts for most of the bill, and it is rarely the one you would guess.