Why is scoring only the final answer a weak way to evaluate agents?
Explain why measuring only final answer accuracy gives an incomplete picture of agent quality. What does trajectory evaluation measure, and what failure modes does it catch that endpoint accuracy misses?
Final answer accuracy ignores how the agent got there. Trajectory evaluation scores the path: per step tool correctness, efficiency, and plan coherence over the whole trace.
Imagine grading a student only by the final number they circle on a math test. Two students both write 42. One solved it cleanly in three lines. The other scribbled for a page, made two mistakes, erased them, and stumbled onto the right answer by luck. The final number is identical, so a pass or fail grader calls them equal. But you would not trust the second student to get the next problem right. Trajectory evaluation is like grading the working out, not just the boxed answer. It reads every step the agent took, checks whether each tool call made sense, counts how many steps it wasted, and asks whether the plan held together. That tells you whether the agent will keep succeeding, or whether it just got lucky this once.
Detailed answer & concept explanation~8 min readEverything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Everything important, quickly.
Open with the destination-versus-journey framing, name lucky recovery and graceful failure as the two blind spots, list the four trajectory metrics, contrast reference based scoring with an LLM judge over the trace, distinguish outcome from process metrics, and close on partial credit plus a benchmark and tracing tool.
| Dimension | Outcome metrics | Process metrics |
|---|---|---|
| What it scores | The final answer only | Every step of the trajectory |
| Example | Task success rate, pass at k | Step efficiency, tool-call correctness, plan coherence |
| Objectivity | Objective, easy to verify | Partly subjective, often judge scored |
| Signal density | Sparse, one bit per run | Dense, one signal per step |
| Catches lucky recovery? | No | Yes |
Real products, models, and research that use this idea.
- SWE-bench evaluates coding agents on whether the generated patch passes the hidden test suite, but practitioners pair that endpoint pass rate with trajectory traces to see how the agent edited, ran tests, and recovered from failures.
- AgentBench scores agents across eight environments and surfaces step level behaviour, so a high task success rate with bloated step counts flags an inefficient policy rather than a strong one.
- LangSmith and Langfuse capture per-step trace spans with tool name, arguments, and observation, which is the raw data a reference based scorer or an LLM judge needs to grade a trajectory.
- Anthropic's evaluation of Claude Opus 4.7 coding agents tracks per-step tool-call validity and retry behaviour alongside final task completion to catch lucky recoveries before release.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you build a partial credit scorer for per-step tool-call correctness against a ground-truth trajectory?
QHow do you stop an LLM judge over the trace from rewarding verbose, confident looking reasoning?
QWhen outcome and process metrics disagree, which do you trust, and what does the gap tell you?
Don't say thisRed flags and common mistakes that signal junior thinking. Click to expand.
Red flags and common mistakes that signal junior thinking. Click to expand.
Treating success rate as the whole story. A high pass rate can hide lucky recoveries and wasteful trajectories that collapse the moment the input shifts slightly.
The night-before-the-interview bullets. Scan these on the way to the call.
Primary sources. Skim if you want the original framing.
Same topic, related formats. Practice these next.