Ask
28

An agent that can call paid APIs in a loop — how do you stop it emptying the account?

I have an agent that can call tools, and some of those tools cost money per call — model inference, a search API, a data provider. It decides how many times to call them.

That is a loop with a budget on the other end of it, and the loop is written by something I cannot fully predict. A bug or a stubborn retry could run all night.

A batch of tools launched recently around exactly this — giving an agent a spending allowance rather than a blank cheque, tracking spend per agent, brokering access to model providers. Before I evaluate any of them, what are the controls that actually work?

4 answers Share
Report

Answering anonymously — a moderator will review it first.

  • @approval_aylin · 4h ago

    One framing that changes what you build: treat the agent as an untrusted client of your own API, not as part of your application.

    You would not let an unknown caller hit a paid endpoint with no quota, no rate limit and no per-key accounting. Give the agent its own credential, its own quota and its own bill, exactly as you would an external integration.

    That is what the newer tools in this space are essentially selling — a credential with an allowance attached rather than your production key. You can build the same thing in an afternoon with a proxy in front of your providers, and the proxy is worth having anyway because it is where the logging, the caching and the kill switch live.

    The kill switch is the part people skip. One flag that makes every tool call fail immediately, reachable without a deploy.

    22
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
  • @token_budget_tuna · 7h ago

    Layer them, because each one fails differently and only the last one is guaranteed.

    1. A hard ceiling at the provider. Every serious API provider offers a spend limit or a prepaid balance. This is the only control that does not depend on your code being correct, so set it first and set it low. Everything else in this list is your software policing your software.

    2. A per-run budget the agent cannot see or change. Not a number in the prompt — a counter in the calling code. Each tool call decrements it; at zero the tool layer refuses and returns an error the agent has to handle. Anything the model can talk itself out of is not a limit.

    3. A per-tool call cap. Separate from cost. "Search may be called eight times in this run" catches the runaway loop long before the money does, and it catches loops on free tools too.

    4. Wall-clock and step limits. The cheapest safety net there is, and it bounds the failure mode where the agent is stuck rather than expensive.

    5. Alerting on rate, not total. A daily total tells you tomorrow. Spend per minute crossing a threshold tells you now.

    30
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
  • @agent_ops_ozan · 4h ago

    The thing I would add from operating these: the expensive failure is almost never a single expensive call. It is a cheap call repeated.

    What that means practically is that your per-call cost estimate barely matters and your loop detection matters enormously. The runs that hurt me were an agent retrying a failing tool with a slightly different argument each time, so no naive duplicate check caught it, several hundred times.

    Two things that helped more than any budget:

    • Detect repetition semantically. Same tool, similar arguments, no progress on the task for N steps → stop. Not a cost control, a sanity control, and it fires earlier.
    • Make failures terminal by default. A tool that fails twice should stay failed for that run rather than remaining available. Most runaway loops are an agent optimistically retrying something that will never work.

    And log cost per run against the run's outcome. You will find a small number of runs consuming most of the spend, and they are usually the ones that failed.

    26
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
  • @evals_efe · 36m ago

    Worth separating two questions that get merged: capping the damage and knowing what you are getting for the money.

    Everything above is the first one. The second one needs cost recorded per run alongside whether the run succeeded, so you can say what a completed task costs rather than what a day costs.

    Once you have that, a surprising number of optimisation arguments resolve themselves — you will usually find one step, or one tool, accounting for most of it, and that the cheaper model is fine for the other nine.

    1
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report