Explain why final-task success rate alone is insufficient for evaluating LLM agents, and describe what trajectory evaluation adds. What specific metrics does trajectory eval provide that enable better debugging of multi-step failures?
Free · 2 AI evals / day
An **LLM agent** solves a task by taking many steps: it plans, calls tools, reads results, and acts again until it finishes. The simplest way to score it is final-task success rate: did the whole task get done, yes or no?
That single number is weak for two reasons. First, it is sparse. Hard agent tasks fail most of the time, so a small set of runs gives you a noisy percentage that bounces around. Second, it is silent about why. If the agent failed a ten-step task, the success number does not tell you whether step two or step seven broke.
Trajectory evaluation fixes this by scoring the intermediate steps, not just the end. It records every action the agent took and grades each one.
Four metrics show up again and again. Tool call accuracy: did the agent pick the right tool with the right arguments? State validity: after each action, is the world in the expected state? Efficiency: how many redundant or wasted actions did it take? Error recovery: after a failed call, did the agent adapt or spiral?
With these, debugging becomes targeted. If tool call accuracy drops at step three across many runs, you know exactly where to look.
Final-task success rate is a binary, end of episode signal: did the agent finish the task or not. It fails as a primary metric for two reasons.
It is sparse and high variance. Hard agent benchmarks fail most of the time, so a handful of runs gives a noisy percentage. Worse, it offers no failure localization. A failed ten-step task tells you nothing about whether step two or step seven broke. Two agents at the same success rate can differ wildly. One may use five clean tool calls; another fifty redundant ones.
**Trajectory evaluation** scores the intermediate steps. It treats the run as a sequence of actions, not one outcome. The standard step-level metrics start with tool call accuracy (right tool, right parameters) and state validity (world state matches the expected state after each action). Then come efficiency (count of redundant or no-op actions) and error recovery (does the agent adapt after a failed call).
These unlock targeted debugging. A drop in tool call accuracy at step three across many trajectories points to a specific planning or prompting failure that a global pass-fail number would mask. You can also award partial credit on a ground-truth trajectory, turning a sparse binary into a graded signal that discriminates between near-misses and total failures.
Quick reference
TL;DR
Success rate is a sparse binary outcome with no failure localization. Trajectory eval scores each step (tool-call validity, state, efficiency, recovery) so you can pinpoint where a multi-step run broke.
Imagine grading a student only by whether they got the final answer right on a long word problem. If they failed, you learn nothing useful: was the setup wrong, did they add wrong, or did they copy the answer wrong at the end? You cannot help them improve. Trajectory evaluation is like grading every line of their working. Now you see exactly which step went off the rails. Maybe they always mess up the same step. With agents this matters because tasks have many steps: search the web, read a file, call an API, write the result. The final 'did it work?' answer hides all of that. Scoring each action tells you the agent is great at searching but keeps calling the wrong tool to save files, so you fix that one thing.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Single-step or single-tool tasks where the agent acts once and stops There is no meaningful trajectory to score, so outcome correctness already captures everything.
Headline leaderboard reporting and cross-team model comparison Outcome success rate is objective and game-resistant; step metrics depend on labels and judges that differ across teams.
Very early prototyping with no labeled reference trajectories or checkpoints Step-level scoring needs ground-truth states or a judge rubric, which are expensive to build before the agent stabilizes.
Tasks with cheap, exact, verifiable end states such as a passing test suite An objective outcome check is unbiased and sufficient; trajectory scoring adds cost without changing the verdict.
Quick scan · 2–4 min
Everything important, quickly.
5 min: why outcome success rate is sparse and unattributable, the four step-level trajectory metrics, ground-truth and partial-credit design, LLM-judge over trajectories, and the outcome as anchor tradeoff.
Real-world examples
Real products, models, and research that use this idea.
1SWE-bench Verified reports task resolution rate, but harnesses like SWE-agent log full action trajectories so failures trace to a specific edit or test step.
2GAIA and AgentBench score multi-step tasks where teams inspect trajectories to localize tool-selection failures rather than trusting the pass rate alone.
3LangSmith and Langfuse trace every agent step (tool, args, observation) so engineers replay trajectories and attach step-level evaluators.
4tau-bench scores agent runs against expected database state transitions, a state-validity check rather than a pure outcome metric.
5OpenAI and Anthropic agent evals pair outcome success with LLM-judge rubrics over the full trajectory for attribution.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QHow do you build ground-truth trajectories when many valid action paths reach the same goal?
Sign in or go Pro to see the approach for this follow-up.
QHow would you award partial credit on a trajectory instead of a binary pass or fail?
Sign in or go Pro to see the approach for this follow-up.
QWhat are the failure modes of using an LLM judge to score a full agent trajectory?
Sign in or go Pro to see the approach for this follow-up.
QHow do you validate a tool call as correct when arguments can be expressed many ways?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Reporting only end to end success rate for an agent. It is sparse, high variance, and tells you nothing about which of ten steps failed, so you cannot debug or improve the agent.