How do you tell whether an agent got better or worse after you changed something?

On the tracing products specifically: what they are genuinely good at is the thing that is annoying to build, which is making one run inspectable after the fact.

When a run fails in production you want the full sequence — prompts, tool calls, arguments, returns, timings — laid out and searchable, months later. Building that yourself is a week and maintaining it is forever.

What they are not is an evaluation set. Several of them offer model-graded scoring, which is useful for things code cannot check — tone, whether a summary is faithful — and it is a fuzzy instrument. Grade with it if you must, and do not let it be your only signal, because a model grader has its own failure modes and they correlate with the model being graded.

So: buy tracing if the storage and UI is worth the money to you. Build the benchmark yourself regardless. The second one is where the answer to your question actually lives.

The move that unlocks everything: evaluate the outcome, not the trajectory.

You are right that there is no single correct path, so scoring the steps is a trap. What you can score is whether the run achieved the thing. For most real tasks that is checkable by code:

• Did the tests pass?
• Does the produced file parse, and contain the fields it was supposed to?
• Did the API call that was supposed to happen, happen, with the right arguments?
• Is the answer's stated figure equal to the known figure?

That gives you a binary or near-binary per case, which is all you need. Build fifty such cases from real tasks you have run, freeze them, and you have a benchmark that survives a prompt rewrite.

Fifty is enough to detect a change of the size you care about. People delay building this because they imagine needing thousands; you do not, because you are comparing two versions on the same set, not estimating an absolute.

One discipline that costs nothing and is worth more than any tool: change one thing at a time, and write down what you expected.

I keep a file with a line per change: what I altered, what I predicted, what the benchmark did. Half of my predictions have been wrong, which is the entire value of writing them down.

Without it you end up with a prompt that has accumulated twelve modifications, all of which somebody believed helped, and no way to remove any of them.

What to record alongside the pass or fail, because the score alone will not tell you what to fix:

• Steps taken and wall-clock. A run that succeeds in three steps and one that succeeds in twenty are not the same result.
• Cost per run.
• Tool call counts, and failed tool calls. The single most diagnostic number I track. A rising failure rate on one tool explains most regressions.
• Where it stopped on the failures, bucketed. Not the transcript — the bucket. "Ran out of steps", "tool errored", "produced malformed output", "gave up".

That last one is what turns a number back into an action. A drop from 80% to 70% is a mystery; a drop concentrated entirely in "produced malformed output" is an afternoon's work.