Flashcard: what is an agent 'trajectory' and why does it matter for evaluation?

Flashcard·Easy·4.0 · 0·~30s

Attempt it

Your answer

**A trajectory** is the full record of everything an agent did during a run. Every thought the model wrote out, every tool call it made, every observation it received back, in the order they happened. If you replayed the trajectory step by step, you would see exactly how the agent went from the user's task to its final answer. The trajectory matters because the final answer alone hides too much. Two agents can both produce the correct answer through completely different paths. One might solve the task in three clean steps; the other might call the wrong tool five times, recover by accident, and stumble onto the right answer after fifteen steps. Final-answer accuracy says they are equally good. Trajectory evaluation says they are not. Production agent evaluation looks at trajectories on three axes. Did the agent call the right tools in the right order? Did it call them with sensible arguments? Did it avoid unnecessary steps and dangerous side effects? Frameworks like Langfuse, LangSmith, Arize Phoenix, and Weave capture trajectories as structured trace data, and benchmarks like SWE-bench, AgentBench, and TAU-bench score them. The short version is: final-answer eval tells you whether the agent got there. Trajectory eval tells you whether it deserves to.

**A trajectory** is the full sequence of thought, action, observation, thought, action, observation steps an agent went through on a single run. It is the agent's equivalent of a profiler trace or a debugger session: every model output, every tool call, every observation, in order, with timestamps and metadata. Trajectory evaluation matters because final-answer accuracy alone misses everything that matters operationally. An agent that lands the right answer in fifteen wandering steps is fragile: on the next slightly different task it will probably wander differently and fail. An agent that lands the right answer in three clean steps is robust: its reasoning was structured and its tool use was efficient. Final accuracy reports them as equally good. Trajectory scoring reports them honestly. The canonical evaluation dimensions are four. **Tool-selection correctness**: did the agent call the right tool at each step, given what it knew. **Argument correctness**: were the arguments well-formed and sensible. **Path efficiency**: how close to the minimum step count did the agent run. **Safety and side-effect correctness**: did the agent avoid state-changing tools it should not have used, and did its retries respect idempotency. Frameworks like Langfuse, LangSmith, Arize Phoenix, Helicone, and Weave capture trajectories as nested trace spans you can inspect. Benchmarks like SWE-bench (real GitHub issues), AgentBench (multi-environment), GAIA (assistant tasks), and TAU-bench (tool-use focused) score both final answers and trajectories. LLM-as-judge graders also work well at trajectory scale, since a human cannot review every step of every run at production scale.

Quick reference

TL;DR

A trajectory is the full thought, action, observation sequence an agent ran through. Trajectory evaluation scores the path, not just the final answer, so you can catch inefficient, unsafe, or accidentally correct runs.

Memory aid

Easy to grasp

Picture two students who both turn in the same correct math answer. One solved it cleanly in three lines. The other wrote two pages of crossed-out attempts, accidentally divided by zero twice, and somehow landed on the right number. As a teacher, do you grade them the same? Probably not. The final answer hides everything you actually care about: did they understand the method, did they avoid dangerous mistakes, would they get the next problem right too. A trajectory is the agent's two pages of work. It is every thought, every tool call, every observation in order. Looking at the trajectory tells you whether the agent solved the task for the right reasons or got there by luck.

Key concepts

Agent Trajectory Agent Evaluation ReAct Observability

Detailed answer & concept explanation

~7 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Quick scan · 2–4 min

Everything important, quickly.

5 min: define trajectory as the ordered thought, action, observation sequence, explain why final-answer eval misses path-level information, list the four canonical scoring axes, name two or three observability tools and two or three benchmarks, and mention LLM-as-judge as the scaling mechanism.

Real-world examples

Real products, models, and research that use this idea.

Langfuse, LangSmith, Arize Phoenix, Helicone, and Weave all capture agent trajectories as nested trace spans; each turn becomes an inspectable call stack with model, tool, and observation spans.
SWE-bench scores real GitHub issue resolutions but exposes the full trajectory so researchers can debug why a model failed on specific repos.
TAU-bench (Tool-Use Agent Benchmark) is explicitly trajectory-focused; tool-call sequence quality, not just final answer, contributes to the score.
AgentBench evaluates trajectories across eight environments (web, code, OS, etc.), exposing per-step latency, cost, and success metrics for cross-task comparison.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QHow do you score thousands of trajectories without reading each one by hand?

QWhat does a trajectory tell you that you cannot see in the final answer alone?

QWhy do production agent teams emit trajectories in OpenTelemetry format?

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Grading agents on final-answer accuracy only. Two agents can both land the right answer with wildly different paths; the wasteful or unsafe one will fail on the next slightly different task.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

References

Primary sources. Skim if you want the original framing.