What does trajectory evaluation add beyond task success rate for LLM agent evaluation?
Trajectory eval scores each tool call and state transition along the agent's run, so you can localize where multi-step agents fail. Success rate only tells you that they failed.
Imagine grading a student on a long math problem. Success rate is like marking the final answer right or wrong, you learn nothing about where they went astray. Trajectory evaluation is like grading every line of work: which formula they picked, whether each step followed, whether they did pointless extra steps. Now if twenty students all slip at line 3, you know exactly what to reteach. For an agent that calls tools across many steps, the final-answer grade hides which tool call was wrong, which step was redundant, and which early mistake the agent later recovered from. Reading the whole trajectory turns a single pass-fail number into a map of failure that tells you what to fix.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
An LLM agent is a loop: observe, decide, act, observe the result, decide again, until the task is done or a budget is exhausted. Each action is usually a tool call (search, code execution, an API request) whose result feeds the next decision. The hard question is how to evaluate something that takes a long, branching path to an answer.
The default metric is task success rate: the fraction of tasks where the final state or answer is correct. It is honest, cheap, and the right top-line number. But it is a single bit per task. For a multi-step agent, one bit throws away almost everything you need to improve the system.
This question asks what trajectory evaluation adds on top, and the answer is step-level diagnostic signal that lets you localize failures. The deep dive below walks the outcome versus process distinction, the specific axes step scoring grades, how aggregation converts step scores into a fixable bug, the two subtle signals that outcome metrics miss, why the distractor options overclaim, and how production stacks combine both metrics.
Outcome metrics versus process metrics
Task success rate is an outcome metric. It answers a single question: did the agent finish the task correctly. That is exactly what you want for a release gate or a leaderboard, because it is unambiguous and hard to game.
Trajectory evaluation is a process metric. It answers a different question: how did the agent get there, and where did the path go wrong. An eight-step task that fails at step five and one that fails at step one have identical outcome scores, yet they are completely different bugs. Outcome metrics cannot tell them apart.
The two metrics are complementary, not competing. The mature workflow uses success rate as the headline number and trajectory eval as the debugger. You ship on the outcome and you fix on the process. Treating either as a replacement for the other is the conceptual error the distractor options are built on.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- SWE-bench grades whether a generated patch passes the repo's tests, while failure analyses inspect the edit trajectory to localize where agents go wrong.
- GAIA tasks require multi-step tool use, and leaderboard analyses break down failures by step type rather than reporting only pass rate.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you score a trajectory when several valid tool-call paths solve the same task?
Move from exact path match to per-step rubric scoring or outcome-conditioned checks: did this step advance a valid sub-goal, regardless of which reference path it matches. An LLM judge with a state-progress rubric handles multi-path tasks better than equality.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Reporting only end to end success rate for a multi-step agent. A single pass or fail number cannot tell you which step broke, so you cannot localize or fix the failure.
60 second bullets to scan on the way to the call.
Difference between outcome metrics and process metrics for agents
What step-level scoring actually grades along a trajectory
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.