What does trajectory evaluation measure that final answer accuracy alone cannot capture?
Trajectory evaluation scores the path (tool choice, step count, plan coherence, recovery), not just the endpoint, so it catches lucky right answers and wasteful or incoherent routes.
Imagine grading a student's maths homework. Final answer accuracy only checks the boxed number at the bottom. If it matches the key, full marks, even if the student guessed or copied a friend. Trajectory evaluation grades the working. It looks at each line: did the student pick the right method, skip pointless steps, and recover sensibly after a mistake? Two students can both write the correct final number. One got there in three clean steps. The other scribbled twenty messy lines and stumbled onto it by luck. Final answer grading gives both the same score. Trajectory grading sees that only one of them actually understands the problem and will get the next one right too. For an agent, the working is its sequence of tool calls and reasoning steps.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Evaluating an agent is harder than evaluating a single model call, because an agent is not one prediction. It is a sequence of decisions: observe, reason, call a tool, read the result, decide the next move, and repeat until done. The output you finally see is the last link in a long chain, and that chain is where almost all of the interesting behaviour lives.
Final answer accuracy scores only that last link. It asks whether the endpoint is correct and ignores everything that produced it. Trajectory evaluation scores the chain itself: the tool calls, their arguments, their ordering, and the coherence of step to step reasoning. It treats the run as evidence about a policy rather than as a single sample.
The two answer different questions. Final answer accuracy asks whether the agent was right this time. Trajectory evaluation asks whether the agent was right for the right reasons, which is what predicts whether it will be right next time. For a single shot classifier those questions collapse into one. For a multi-step agent they come apart, and the gap between them is exactly the space where production failures hide.
Why the endpoint hides the failure
An agent can reach a correct final answer through a process you would never want to ship. Three patterns recur, and endpoint accuracy is blind to all of them.
The first is the lucky recovery. The agent picks a wrong tool, gets a confusing result, flails, and then stumbles onto the right answer anyway. The endpoint is correct, but the path is not repeatable. Change the input slightly and the luck evaporates. On a benchmark with a fixed seed this agent looks strong; on the next real ticket it collapses, and you have no early warning because the leaderboard only recorded the win.
The second is the wasteful detour. The agent solves the task but takes thirty tool calls where five would do. Final answer accuracy gives it full marks while it quietly triples your token bill and your latency. In production, a slow correct path can still be too expensive to keep, and a thirty step run also has thirty chances to trip over a transient tool error that a five step run never faces.
The third is the incoherent path. Steps do not follow from the prior observation, the agent re-fetches data it already has, or it ignores a tool error and proceeds on stale assumptions. The answer happens to land correct, but the reasoning is broken and will fail on a harder variant. Each of these three agents posts the same perfect endpoint score, which is precisely why you need a second axis of measurement to tell them apart.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- tau-bench scores customer service agents on both task success and adherence to the correct tool sequence, exposing agents that pass on outcome but violate policy on the path.
- SWE-bench measures whether a coding agent's final patch passes the hidden test suite, while SWE-bench Verified and trace tooling let teams inspect the edit test trajectory behind each pass.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you score a trajectory when many valid step sequences solve the same task?
Drop exact match in favour of property based checks and rubric scoring. Assert invariants the path must hold, grade coherence and efficiency with an LLM judge against a rubric, and reserve strict reference matching only for safety critical ordering constraints.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Assuming a correct final answer means a correct process. An agent can reach the right output through a lucky recovery or a wasteful path that breaks on the next input.
60 second bullets to scan on the way to the call.
State the one thing final answer accuracy measures and the one thing it cannot see.
Name the four trajectory signal families and what each one catches.
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.