Why is scoring only the final answer a weak way to evaluate agents?

Attempt it

Explain why measuring only final answer accuracy gives an incomplete picture of agent quality. What does trajectory evaluation measure, and what failure modes does it catch that endpoint accuracy misses?

Free · 2 AI evals / day

Quick reference

TL;DR

Final answer accuracy ignores how the agent got there. Trajectory evaluation scores the path: per step tool correctness, efficiency, and plan coherence over the whole trace.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Imagine grading a student only by the final number they circle on a math test. Two students both write 42. One solved it cleanly in three lines. The other scribbled for a page, made two mistakes, erased them, and stumbled onto the right answer by luck. The final number is identical, so a pass or fail grader calls them equal. But you would not trust the second student to get the next problem right. Trajectory evaluation is like grading the working out, not just the boxed answer. It reads every step the agent took, checks whether each tool call made sense, counts how many steps it wasted, and asks whether the plan held together. That tells you whether the agent will keep succeeding, or whether it just got lucky this once.

Key concepts

Agent Evaluation Trajectory Evaluation Tool-Use Correctness LLM as Judge

Quick scan · 2–4 min

Everything important, quickly.

Open with the destination-versus-journey framing, name lucky recovery and graceful failure as the two blind spots, list the four trajectory metrics, contrast reference based scoring with an LLM judge over the trace, distinguish outcome from process metrics, and close on partial credit plus a benchmark and tracing tool.

Side-by-side

Dimension	Outcome metrics	Process metrics
What it scores	The final answer only	Every step of the trajectory
Example	Task success rate, pass at k	Step efficiency, tool-call correctness, plan coherence
Objectivity	Objective, easy to verify	Partly subjective, often judge scored
Signal density	Sparse, one bit per run	Dense, one signal per step
Catches lucky recovery?	No	Yes

Real-world examples

Real products, models, and research that use this idea.

SWE-bench evaluates coding agents on whether the generated patch passes the hidden test suite, but practitioners pair that endpoint pass rate with trajectory traces to see how the agent edited, ran tests, and recovered from failures.
AgentBench scores agents across eight environments and surfaces step level behaviour, so a high task success rate with bloated step counts flags an inefficient policy rather than a strong one.
LangSmith and Langfuse capture per-step trace spans with tool name, arguments, and observation, which is the raw data a reference based scorer or an LLM judge needs to grade a trajectory.
Anthropic's evaluation of Claude Opus 4.7 coding agents tracks per-step tool-call validity and retry behaviour alongside final task completion to catch lucky recoveries before release.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QHow would you build a partial credit scorer for per-step tool-call correctness against a ground-truth trajectory?

Sign in or go Pro to see the approach for this follow-up.

QHow do you stop an LLM judge over the trace from rewarding verbose, confident looking reasoning?

Sign in or go Pro to see the approach for this follow-up.

QWhen outcome and process metrics disagree, which do you trust, and what does the gap tell you?

Sign in or go Pro to see the approach for this follow-up.

QHow do you separate agent-caused failures from environment caused failures in your metrics?

Sign in or go Pro to see the approach for this follow-up.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.