Why is final task success rate insufficient for evaluating LLM agents, and what does trajectory eval add?
Explain why final task success rate alone is insufficient for evaluating LLM agents, and describe what trajectory evaluation adds. What specific metrics does trajectory eval provide that enable better debugging of multi-step failures?
Success rate is a sparse binary outcome with no failure localization. Trajectory eval scores each step (tool-call validity, state, efficiency, recovery) so you can pinpoint where a multi-step run broke.
Imagine grading a student only by whether they got the final answer right on a long word problem. If they failed, you learn nothing useful: was the setup wrong, did they add wrong, or did they copy the answer wrong at the end? You cannot help them improve. Trajectory evaluation is like grading every line of their working. Now you see exactly which step went off the rails. Maybe they always mess up the same step. With agents this matters because tasks have many steps: search the web, read a file, call an API, write the result. The final 'did it work?' answer hides all of that. Scoring each action tells you the agent is great at searching but keeps calling the wrong tool to save files, so you fix that one thing.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
LLM agents do not produce one answer; they run an episode. The agent observes a state, plans, calls a tool, reads the observation, and loops until it declares the task done. Evaluating that episode is fundamentally harder than scoring a one-shot completion, and the gap between the naive metric and a useful one is exactly what senior agent-eval questions probe.
The naive metric is final-task success rate: a single bit at the end of the episode. It is attractive because it is objective, cheap, and matches what the user actually cares about. The user does not reward an agent for an elegant plan that fails; they reward a finished task. So success rate is the right thing to put on a leaderboard. The problem is that it is the wrong thing to engineer against, because it carries almost no information about how the agent behaved on the way to that final bit.
This deep dive walks why outcome metrics fail, what trajectory evaluation measures instead, the hard problems of ground truth and partial credit, the role of an LLM judge over trajectories, and the production tradeoff that keeps outcome success as the anchor while trajectory metrics do the diagnostic work.
Why outcome success rate is not enough
Final-task success rate has two structural weaknesses. It is sparse, and it is unattributable.
Sparsity is a statistics problem. Hard agent suites resolve a minority of tasks, so on a few hundred runs the pass rate is a noisy estimate with wide confidence intervals. A one-point move between model versions can be pure variance. Worse, the signal lives only on the tasks that flipped from fail to pass, so most of your evaluation data carries no gradient about whether the agent got better.
Unattributability is the debugging problem. A ten-step task that fails gives you one bit: it failed. It does not tell you the agent planned correctly but called a tool with a malformed argument at step three, or that it recovered from that and then looped forever at step eight. Two agents can share an identical success rate while one runs five clean tool calls and the other fifty redundant ones. The metric cannot see the difference, so it cannot guide an improvement.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- SWE-bench Verified reports task resolution rate, but harnesses like SWE-agent log full action trajectories so failures trace to a specific edit or test step.
- GAIA and AgentBench score multi-step tasks where teams inspect trajectories to localize tool-selection failures rather than trusting the pass rate alone.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow do you build ground-truth trajectories when many valid action paths reach the same goal?
Avoid one canonical path. Score against path-agnostic state checkpoints or a set of reference trajectories, or use an LLM judge with a rubric over the whole run. Match on semantic equivalence of tool arguments, not string equality.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Reporting only end to end success rate for an agent. It is sparse, high variance, and tells you nothing about which of ten steps failed, so you cannot debug or improve the agent.
60 second bullets to scan on the way to the call.
Why success rate is sparse and high variance on hard agent suites
Why success rate gives no failure localization
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.