Efe

Worth separating two questions that get merged: capping the damage and knowing what you are getting for the money.

Everything above is the first one. The second one needs cost recorded per run alongside whether the run succeeded, so you can say what a completed task costs rather than what a day costs.

Once you have that, a surprising number of optimisation arguments resolve themselves — you will usually find one step, or one tool, accounting for most of it, and that the cheaper model is fine for the other nine.

What to record alongside the pass or fail, because the score alone will not tell you what to fix:

• Steps taken and wall-clock. A run that succeeds in three steps and one that succeeds in twenty are not the same result.
• Cost per run.
• Tool call counts, and failed tool calls. The single most diagnostic number I track. A rising failure rate on one tool explains most regressions.
• Where it stopped on the failures, bucketed. Not the transcript — the bucket. "Ran out of steps", "tool errored", "produced malformed output", "gave up".

That last one is what turns a number back into an action. A drop from 80% to 70% is a mystery; a drop concentrated entirely in "produced malformed output" is an afternoon's work.

The under-discussed cost: tool descriptions are prompt, and prompt is a budget.

Every tool your server exposes puts its name, description and argument schema into the model's context on every turn. Twenty tools with generous descriptions is a meaningful slice of the window before the user has said anything, and it measurably degrades selection accuracy — models get worse at choosing when the menu is long.

So a server exposing forty operations because the REST API has forty endpoints is actively harmful. The good ones expose a small number of task-shaped tools, not a mechanical mirror of an API surface.

That is the design work, and it is the part that mechanical wrapper generators skip. If somebody proposes generating a server from your OpenAPI spec, that is the failure mode to argue against.

The move that unlocks everything: evaluate the outcome, not the trajectory.

You are right that there is no single correct path, so scoring the steps is a trap. What you can score is whether the run achieved the thing. For most real tasks that is checkable by code:

• Did the tests pass?
• Does the produced file parse, and contain the fields it was supposed to?
• Did the API call that was supposed to happen, happen, with the right arguments?
• Is the answer's stated figure equal to the known figure?

That gives you a binary or near-binary per case, which is all you need. Build fifty such cases from real tasks you have run, freeze them, and you have a benchmark that survives a prompt rewrite.

Fifty is enough to detect a change of the size you care about. People delay building this because they imagine needing thousands; you do not, because you are comparing two versions on the same set, not estimating an absolute.

Add a second principle that comes from measuring it: the reconciliation cost scales with how much they had to agree on.

If the outputs are independent artefacts — a summary each, a file each — merging is trivial. If the outputs are opinions about the same design, merging is a meeting.

So before splitting, ask what has to be true for the pieces to fit together, and how much of that is written down. If the answer is "it is in my head", the agents will each invent a different version of it and you have created work.

The practical version: make the contract explicit before the fan-out. Interfaces, file boundaries, naming, what each is responsible for. Doing that costs ten minutes and it is exactly the ten minutes that decides whether the split helps.