Flashcard: what is an agent 'trajectory' and why does it matter for evaluation?

Flashcard·Easy·4.0 · 0·~30s·Asked atFractal Analytics Niki Ai Robinhood·Relevant atDatabricks

Attempt it

Your answer

**A trajectory** is the full record of everything an agent did during a run. Every thought the model wrote out, every tool call it made, every observation it received back, in the order they happened. If you replayed the trajectory step by step, you would see exactly how the agent went from the user's task to its final answer. The trajectory matters because the final answer alone hides too much. Two agents can both produce the correct answer through completely different paths. One might solve the task in three clean steps; the other might call the wrong tool five times, recover by accident, and stumble onto the right answer after fifteen steps. Final-answer accuracy says they are equally good. Trajectory evaluation says they are not. Production agent evaluation looks at trajectories on three axes. Did the agent call the right tools in the right order? Did it call them with sensible arguments? Did it avoid unnecessary steps and dangerous side effects? Frameworks like Langfuse, LangSmith, Arize Phoenix, and Weave capture trajectories as structured trace data, and benchmarks like SWE-bench, AgentBench, and TAU-bench score them. The short version is: final-answer eval tells you whether the agent got there. Trajectory eval tells you whether it deserves to.

**A trajectory** is the full sequence of thought, action, observation, thought, action, observation steps an agent went through on a single run. It is the agent's equivalent of a profiler trace or a debugger session: every model output, every tool call, every observation, in order, with timestamps and metadata. Trajectory evaluation matters because final-answer accuracy alone misses everything that matters operationally. An agent that lands the right answer in fifteen wandering steps is fragile: on the next slightly different task it will probably wander differently and fail. An agent that lands the right answer in three clean steps is robust: its reasoning was structured and its tool use was efficient. Final accuracy reports them as equally good. Trajectory scoring reports them honestly. The canonical evaluation dimensions are four. **Tool-selection correctness**: did the agent call the right tool at each step, given what it knew. **Argument correctness**: were the arguments well-formed and sensible. **Path efficiency**: how close to the minimum step count did the agent run. **Safety and side-effect correctness**: did the agent avoid state-changing tools it should not have used, and did its retries respect idempotency. Frameworks like Langfuse, LangSmith, Arize Phoenix, Helicone, and Weave capture trajectories as nested trace spans you can inspect. Benchmarks like SWE-bench (real GitHub issues), AgentBench (multi-environment), GAIA (assistant tasks), and TAU-bench (tool-use focused) score both final answers and trajectories. LLM-as-judge graders also work well at trajectory scale, since a human cannot review every step of every run at production scale.

TL;DR

A trajectory is the full thought, action, observation sequence an agent ran through. Trajectory evaluation scores the path, not just the final answer, so you can catch inefficient, unsafe, or accidentally correct runs.

Memory aid

Easy to grasp

Picture two students who both turn in the same correct math answer. One solved it cleanly in three lines. The other wrote two pages of crossed-out attempts, accidentally divided by zero twice, and somehow landed on the right number. As a teacher, do you grade them the same? Probably not. The final answer hides everything you actually care about: did they understand the method, did they avoid dangerous mistakes, would they get the next problem right too. A trajectory is the agent's two pages of work. It is every thought, every tool call, every observation in order. Looking at the trajectory tells you whether the agent solved the task for the right reasons or got there by luck.

Key concepts

Agent Trajectory Agent Evaluation ReAct Observability

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Trajectory is one of those vocabulary words that sounds technical and turns out to mean something quite simple: the record of everything the agent did. The reason it matters is that LLM agents have far more variance in how they reach an answer than in whether they reach one. Two agents that both pass a benchmark on final-answer accuracy can have completely different operational properties: one is cheap and reliable, the other is expensive and lucky.

This deep dive defines what a trajectory is, explains why final-answer evaluation alone misses the things that matter most in production, walks through the four canonical scoring dimensions, and surveys the 2026 stack for capturing and scoring trajectories. By the end, the case for trajectory-first agent evaluation should be obvious enough that final-only evaluation looks like a beginner's mistake.

What a trajectory actually contains

A trajectory is the ordered sequence of every step the agent ran through on a single execution. Each step contains the model's output (including any reasoning text), the tool call the model emitted (if any), the arguments to that call, the observation the runtime fed back, and timing and cost metadata. Add the user task at the start and the final answer or stopping reason at the end, and you have the full record.

The granularity matters. A minimal trajectory captures only the model calls and final answer; a useful trajectory captures the tool calls and observations too. Without the observations, you cannot tell why the model made the choices it made on each subsequent turn. The ReAct pattern made this granularity explicit by interleaving thoughts and actions in the model's output; the trajectory is essentially the ReAct trace plus runtime metadata.

Modern observability stacks emit trajectories as nested trace spans. The top-level span is the agent run; nested under it are per-turn spans; nested under those are the model-call span and the tool-call spans for that turn. OpenTelemetry has become the standard format, which means Langfuse, LangSmith, Phoenix, Helicone, and Weave can all consume the same emitted traces and let teams swap observability backends without re-instrumenting.

Why final-answer accuracy is not enough

The four canonical scoring dimensions

Capturing trajectories: the 2026 observability stack

Benchmarks and scaling trajectory evaluation

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Real products, models, and research that use this idea.

Langfuse, LangSmith, Arize Phoenix, Helicone, and Weave all capture agent trajectories as nested trace spans; each turn becomes an inspectable call stack with model, tool, and observation spans.
SWE-bench scores real GitHub issue resolutions but exposes the full trajectory so researchers can debug why a model failed on specific repos.

What an interviewer would ask next. Try answering before peeking at the approach.

QHow do you score thousands of trajectories without reading each one by hand?

Combine programmatic rules (cheap, deterministic), LLM-as-judge (broad, noisier), and golden-trace comparison (precise where curated). Run programmatic rules at full scale, sample for LLM judging, reserve golden traces for regression tests.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Grading agents on final-answer accuracy only. Two agents can both land the right answer with wildly different paths; the wasteful or unsafe one will fail on the next slightly different task.

60 second bullets to scan on the way to the call.

Define a trajectory as the ordered sequence of thoughts, actions, and observations.
Explain why final-answer evaluation alone is insufficient.

Primary sources. Browse if you want the original framing.