Ask
26

Stopping an agent that is already running: what actually works when there is no button

An agent went into a loop with real side effects and I discovered there was nothing to press. What I have in place now, in order of how reliably it works.

Things that stop it

  • A hard step and token ceiling per run. Not a safety feature so much as a floor under the damage, and the only one that works while you are asleep. Set it lower than feels necessary.
  • A spend limit at the provider, with an alert well below it. The alert is the part that matters, because it arrives before the ceiling does.
  • Killing the process or the container. Reliable, blunt, and the reason to run agents in something you can destroy without thinking. If the agent runs inside your normal shell, this is a much worse moment.
  • Revoking the credential it is using. The most effective interruption available, because it stops the effects rather than the loop. Give every agent its own key so that revoking one does not take everything else down.

Things that do not

  • Asking it to stop, which is a message in a queue it may not read.
  • A stop button in a wrapper, unless you know it cancels the underlying request rather than the display of it. Several stop the stream and let the run continue.

The structural point: interruption is unreliable and containment is not. Scoped credentials, a throwaway execution environment, and no write access to anything you cannot restore means a runaway run is expensive rather than dangerous. Design for that instead of for a button.

Log what it did, not just what it said. Recovering from a bad run needs the tool calls and their arguments, and most setups keep only the conversation.

8 answers Share
Report

Answering anonymously, a moderator will review it first.

  • @budget_as_brake · 3w ago · 3 replies

    Limits over buttons, agreed. The specific setting people miss is a per-run ceiling as opposed to a monthly one.

    A monthly budget does not stop a single runaway run, it just tells you afterwards. A per-run step limit stops the loop at a fixed cost and turns an incident into an annoyance.

    Set it to roughly twice what a normal run takes. If a run needs more than that, you want to know, and finding out by being interrupted is much better than finding out from a bill.

    24
    Share
    Reply

    Answering anonymously, a moderator will review it first.

    Report
    • @killed_it_with_ctrlc · 2w ago · 2 replies

      The part people discover during the incident: stopping the process does not stop the side effects that are already in flight. Requests already sent complete, queued jobs still run, and anything with a retry keeps going without the agent.

      So the ceiling matters more than the interrupt, because the interrupt is always late by however long your slowest in-flight call is.

      22
      Share
      Reply

      Answering anonymously, a moderator will review it first.

      Report
      • @budget_as_brake · 2w ago

        Which is the argument for the per-run ceiling being a hard stop in the tool layer rather than a check in the loop.

        13
        Share
        Reply

        Answering anonymously, a moderator will review it first.

        Report
  • @sandbox_first · 3w ago · 2 replies

    Containment beating interruption is the thing I would put first rather than last.

    Once an agent runs in a throwaway container with its own scoped credential and no path to anything unrecoverable, the question stops being how fast you can stop it. A bad run costs money and time and nothing else, and you can afford to be asleep.

    The credential scoping is the half people skip. A single shared key means revoking it during an incident takes down everything else that was using it, so you hesitate, and hesitation is the whole problem.

    19
    Share
    Reply

    Answering anonymously, a moderator will review it first.

    Report
    • @killed_it_with_ctrlc · 2w ago

      Scoped throwaway credentials in a container is the only thing on the list that limits damage rather than duration.

      11
      Share
      Reply

      Answering anonymously, a moderator will review it first.

      Report
  • @dry_run_default · 3w ago

    Adding the cheapest control: make destructive tools require an explicit flag and default to a dry run. An agent that must ask for the real version will show you its intent in the log before it does anything, and most loops die harmlessly against a dry run.

    It costs one parameter per tool and it has saved me twice.

    18
    Share
    Reply

    Answering anonymously, a moderator will review it first.

    Report
  • @stop_the_agent · 3w ago · 2 replies

    Logging the tool calls and their arguments rather than only the conversation is the one I learned the hard way. Cleaning up after a bad run needs to know exactly what it did, and a transcript of what it said is not that.

    1
    Share
    Reply

    Answering anonymously, a moderator will review it first.

    Report
    • @dry_run_default · 3w ago

      Log the arguments, not just the call. Cleanup needs to know which record, not that something was written.

      8
      Share
      Reply

      Answering anonymously, a moderator will review it first.

      Report