Why a multi-agent eval has to score the trajectory, not only the final output

Flashcard·Medium·4.0 · 0·~30s·Asked atElevenlabs Krutrim Unity

Attempt it

Two systems can produce the same final answer through wildly different trajectories. One did it in three hops with one tool call; the other took eleven hops, two retries, three handoff loops, and burned six times the budget. Final-output evaluation rates them identically, which hides cost regressions, tool-call errors, and brittle paths that only happened to land correct this time.

Trajectory scoring evaluates the path as well as the destination. Metrics that matter on the path: number of hops, per-agent step-correctness with attribution (which agent introduced the wrong fact, the wrong tool call, the wrong handoff), tool-call precision and recall, coverage of branching paths in the topology graph, total cost, and total wall-clock latency. The implementation surface is heavier (you need to capture the full trace, not just the response) but the eval that comes out actually tells you whether the system is improving or just getting lucky.

Your answer

**Why final-output eval is not enough.** Imagine two multi-agent runs that both produce the right answer. Run A: three hops, one tool call, $0.05 in tokens, two seconds. Run B: eleven hops, two retries, three handoff loops, $0.30 in tokens, fifteen seconds. A final-output eval calls them equal. They are not. **What trajectory eval scores instead.** - **Hop count**: how many supervisor and worker turns the run took. - **Per-agent step-correctness**: did each agent's individual output check out? Which agent introduced the wrong fact? - **Tool-call precision and recall**: did the right tools get called, with the right arguments, in the right order? - **Path coverage**: are the branches of the topology being exercised, or are 95 percent of traces going through one happy path? - **Cost per trace**: total tokens times price. - **Latency per trace**: wall-clock end to end. **Why it matters.** Final-output eval misses cost regressions (a 2x bill increase while accuracy is flat), tool errors that retries papered over, and brittle runs that happened to land correct this time but will fail on the next prompt variation. The implementation cost is higher because you need to capture the full trace, but the eval result actually tells you whether the system is improving or just getting lucky.

**Final-output eval is single-agent thinking.** A multi-agent run produces a final answer through a sequence of intermediate steps: routing decisions, tool calls, handoffs, retries, partial outputs. Two runs can produce the same final answer through wildly different sequences. The cheap, fast, three-hop run and the expensive, slow, eleven-hop run score identically on accuracy. That equivalence hides cost regressions, masks tool-call errors that retries papered over, and rewards systems that happened to land correct this time but will fail on the next prompt variation. **Trajectory eval grades the path, not just the destination.** Six metrics matter: - **Hop count**: shorter is generally better; track distribution and tail. - **Per-agent step-correctness with attribution**: which agent introduced the bug, the wrong fact, the wrong handoff? - **Tool-call precision and recall**: were the right tools called with right arguments, were any missed? - **Path coverage in the topology graph**: are branches exercised, or is the eval only walking one happy path? - **Cost per trace**: tokens times price. - **Latency per trace**: wall-clock end to end. **The implementation cost.** You need to capture the full trace, not just the response. OpenTelemetry GenAI spans, LangSmith or Langfuse or Phoenix as the trace store, and an eval harness that walks the trace tree. **The payoff.** You can tell the difference between 'the system got smarter' and 'the system got lucky.' You can catch cost regressions before the bill arrives. You can attribute failures to a specific agent and fix the right thing.

TL;DR

Final-output eval hides cost regressions, tool errors, and lucky-path runs in multi-agent systems; trajectory eval scores the path itself (hops, per-agent step-correctness, tool precision, cost, latency).

Memory aid

Easy to grasp

Imagine grading a road trip from New York to Boston by only checking that the car arrived. Two trips both arrived. One drove the highway in four hours with one gas stop. The other took back roads for fourteen hours, ran out of gas twice, and got lost in Hartford. They both pass a 'did you arrive' test. The trip with fourteen hours and two gas refills is not the same as the four-hour one, and pretending it is means you never notice when the fleet is getting slower and more expensive. Trajectory eval grades how the car got there, not just whether it arrived.

Key concepts

Multi-Agent Systems

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Multi-agent eval is one of the places single-agent intuition fails most obviously. Single-agent eval is a black-box function: input goes in, output comes out, compare to a golden reference. Multi-agent is a sequence of internal decisions (routing, handoffs, tool calls, retries) that aggregate to a final answer. Two runs can land the same answer through wildly different sequences. The cheap, three-hop, one tool call run and the expensive, eleven-hop, three-retry run are not equivalent, but they are equivalent under final-output eval.

This walkthrough explains why trajectory eval is the right shape for multi-agent, names the six metrics that actually catch problems, and gives concrete implementation guidance: how to capture the trace, how to attribute step-correctness, how to compute path coverage, and how to wire the result into a regression dashboard.

Mental model: for multi-agent, the trajectory IS the system; evaluating only the final output is like grading a road trip by checking the car arrived.

Why final-output eval breaks for multi-agent

The equivalence problem

Final-output eval treats the run as a black box. Two runs that produce the same correct answer are scored equally. But the runs are not equivalent in any production-meaningful way:

The 3-hop run with one tool call costs $0.05 and takes 2 seconds.
The 11-hop run with two retries and three handoff loops costs $0.30 and takes 15 seconds.

If your eval reports 91 percent accuracy and your bill triples month over month, the eval is not helping you debug the regression.

The lucky-path problem

A brittle multi-agent run can land on the correct answer for one specific input by accident. The same system fails on the next prompt variation because the lucky path was not robust. Final-output eval on a held-out set rates the system as 'works most of the time' without telling you that the success was structurally fragile.

The attribution problem

When a multi-agent system gets the wrong answer, the failure happened somewhere inside. Which agent? Which tool call? Which handoff? Final-output eval says 'system was wrong.' That is not actionable. Trajectory eval says 'agent X called tool Y with wrong arguments at hop 4.' That is actionable.

The cost regression problem

A model upgrade, a prompt change, a new tool can all silently increase cost per trace without changing accuracy. Final-output eval is blind to this. Trajectory eval surfaces it immediately because cost per trace is one of the six metrics.

The six trajectory metrics

Implementation: how to actually capture and compute this

The payoff: falsifiable improvement claims

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Real products, models, and research that use this idea.

LangSmith's trace eval explicitly supports per-step grading and per-agent attribution for LangGraph runs.
Langfuse and Arize Phoenix both expose per-span queries that let you aggregate cost per `agent.role` for trajectory eval.

What an interviewer would ask next. Try answering before peeking at the approach.

QHow would you build a path-coverage metric concretely?

Statically enumerate the edges of the topology graph (handoff edges in LangGraph or AutoGen). Tag each trace with the set of edges it traversed. Coverage = unique edges hit by the eval set divided by total edges. Aim for 80 percent plus; below 50 percent means most branches are untested.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Reporting end to end accuracy as the only eval metric and missing that the system got 4x more expensive and brittle over the last release while quality numbers held flat.

60 second bullets to scan on the way to the call.

Why is final output eval insufficient for a multi-agent system?
What are the six trajectory metrics and what does each catch?

Primary sources. Browse if you want the original framing.