What does trajectory evaluation measure that final answer accuracy alone cannot capture?
Trajectory evaluation scores the path (tool choice, step count, plan coherence, recovery), not just the endpoint, so it catches lucky right answers and wasteful or incoherent routes.
Imagine grading a student's maths homework. Final answer accuracy only checks the boxed number at the bottom. If it matches the key, full marks, even if the student guessed or copied a friend. Trajectory evaluation grades the working. It looks at each line: did the student pick the right method, skip pointless steps, and recover sensibly after a mistake? Two students can both write the correct final number. One got there in three clean steps. The other scribbled twenty messy lines and stumbled onto it by luck. Final answer grading gives both the same score. Trajectory grading sees that only one of them actually understands the problem and will get the next one right too. For an agent, the working is its sequence of tool calls and reasoning steps.
Detailed answer & concept explanation~8 min readEverything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Everything important, quickly.
Define final answer accuracy and its blind spot, then name the four trajectory signal families, explain why outcome and process decorrelate, contrast reference matching with rubric judging, and close with the non uniqueness problem and when a strict path check is actually justified.
Real products, models, and research that use this idea.
- tau-bench scores customer service agents on both task success and adherence to the correct tool sequence, exposing agents that pass on outcome but violate policy on the path.
- SWE-bench measures whether a coding agent's final patch passes the hidden test suite, while SWE-bench Verified and trace tooling let teams inspect the edit test trajectory behind each pass.
- AgentBench reports per environment success, but its traces reveal step count and tool misuse patterns that separate a frontier model like Claude Opus 4.7 from a weaker one even at similar success rates.
- LangSmith and Langfuse capture per-step spans so teams can run rubric-based LLM-as-judge over the full tool-call trajectory, not just the final agent message.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you score a trajectory when many valid step sequences solve the same task?
QHow do you combine an outcome reward with a process signal without one drowning the other?
Don't say thisRed flags and common mistakes that signal junior thinking. Click to expand.
Red flags and common mistakes that signal junior thinking. Click to expand.
Assuming a correct final answer means a correct process. An agent can reach the right output through a lucky recovery or a wasteful path that breaks on the next input.
The night-before-the-interview bullets. Scan these on the way to the call.
Primary sources. Skim if you want the original framing.
Same topic, related formats. Practice these next.