An LLM **agent** solves a task by taking many steps: it reads a goal, picks a tool, calls it, reads the result, decides the next move, and repeats until done. The simplest way to grade it is task success rate, the fraction of tasks where the final answer is correct. That number is honest but shallow.
Trajectory evaluation grades the path, not just the destination. It scores each step along the way: was the right tool chosen, were the arguments valid, did the state change as expected, were there pointless extra steps. The answer is option A.
The value is diagnostic. If a hundred tasks fail and you only have success rate, you know the agent is broken but not where. If you score the trajectory and find that 80 percent of failures share a wrong tool choice at step 3, you have found a specific planning bug to fix.
The other options describe things trajectory eval does not do. It does not auto-write failure explanations, does not read the model's internal confidence, and averaging step scores is not always more reliable than the binary success number.
**Trajectory evaluation** scores the agent's sequence of actions, not just its final answer. The correct option is A: step-level scoring of tool calls, state transitions, and efficiency.
Task success rate is a single binary signal per task. It tells you whether the agent finished, but not where it went wrong. For a multi-step agent that hides everything that matters for debugging. An eight-step task that fails at step five looks identical to one that fails at step one.
Trajectory eval opens up the run. It checks step correctness (right tool, right arguments) and state transitions (did the world change as intended). It also flags redundant steps (wasted calls that inflate cost) and recoverable errors (a wrong step the agent later corrected). Aggregated across many runs, this localizes failures. If most failed tasks share an error at the same step, that is a targeted fix.
The distractors are wrong because they overclaim. Trajectory eval does not auto-generate explanations, does not read internal confidence, and a continuous average of step scores is not always more reliable than binary success.
Quick reference
TL;DR
Trajectory eval scores each tool call and state transition along the agent's run, so you can localize where multi-step agents fail. Success rate only tells you that they failed.
Imagine grading a student on a long math problem. Success rate is like marking the final answer right or wrong, you learn nothing about where they went astray. Trajectory evaluation is like grading every line of work: which formula they picked, whether each step followed, whether they did pointless extra steps. Now if twenty students all slip at line 3, you know exactly what to reteach. For an agent that calls tools across many steps, the final-answer grade hides which tool call was wrong, which step was redundant, and which early mistake the agent later recovered from. Reading the whole trajectory turns a single pass-fail number into a map of failure that tells you what to fix.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Single-step or single-tool tasks where the agent makes one decision and stops There is no multi-step path to diagnose, so binary success captures everything trajectory scoring would.
Early prototyping where you only need a coarse go or no-go signal on the whole system Success rate is faster and cheaper to compute, and step rubrics add overhead before the agent even works end to end.
Open-ended tasks with many equally valid solution paths and no agreed reference Step-level scoring against a single golden trajectory penalizes correct alternative paths and produces misleading diagnostics.
High-volume production monitoring where per-step labeling cost is prohibitive Full trajectory scoring on every request is expensive. Sample trajectories for deep eval and track success rate online.
Quick scan · 2–4 min
Everything important, quickly.
4 min: outcome vs process metrics, the four step-level signals (correctness, state, redundancy, recovery), how aggregation localizes a bug, why each distractor overclaims, and the production pattern of gating on success and debugging on the trajectory.
Real-world examples
Real products, models, and research that use this idea.
1SWE-bench grades whether a generated patch passes the repo's tests, while failure analyses inspect the edit trajectory to localize where agents go wrong.
2GAIA tasks require multi-step tool use, and leaderboard analyses break down failures by step type rather than reporting only pass rate.
3LangSmith ships trajectory evaluators that score agent runs against reference tool-call sequences and flag missing or extra steps.
4The OpenAI Agents SDK exposes per-step traces (tool calls, arguments, outputs) so teams can score and replay individual decision points.
5AgentBench reports task success alongside step-level execution traces across web, OS, and database environments to diagnose failure modes.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you score a trajectory when several valid tool-call paths solve the same task?
Sign in or go Pro to see the approach for this follow-up.
QHow do you distinguish a recoverable error from a fatal one in trajectory scoring?
Sign in or go Pro to see the approach for this follow-up.
QWhen does end to end success rate diverge most sharply from step-level trajectory quality?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Reporting only end to end success rate for a multi-step agent. A single pass or fail number cannot tell you which step broke, so you cannot localize or fix the failure.