List the minimum set of fields that every agent execution span should capture. Explain why omitting any one of them makes production debugging significantly harder.
Free · 2 AI evals / day
An agent does not answer in one shot. It loops, taking many small steps, where each step feeds the next. To debug a failed run, you need a record of every step, not just the final answer. That record is a **trace**, and each step is one span inside it.
Every span should capture six things. First, the input context, meaning the exact prompt the model saw at that step. Second, the full model response, including the reasoning, not just the chosen tool call. Third, the tool inputs, the exact arguments the agent passed. Fourth, the tool output or error, whatever came back. Fifth, the latency, how long the step took. Sixth, the cost, the token count for that step.
Why all six? Because when step twelve of a twenty-step run produces a wrong answer, you need to see what the agent was looking at, what it decided, what it tried, and what came back. Drop any field and a whole class of bugs becomes a guessing game.
The big idea is that the model's reasoning vanishes the moment the turn ends. If you did not write it down, it is gone, and the failure is no longer reproducible.
An agent run is a sequence of dependent steps, so you trace it as one span per step, not one trace per request. Each span must capture six fields.
The six are the input context snapshot sent to the model, the full model response including its reasoning trace, and the tool arguments. The other three are the tool output or error, the step latency, and the step cost in tokens.
Each field unlocks a class of bug. The input snapshot makes the step reproducible. The reasoning trace explains why the model chose an action. Logging tool inputs separately from the model response lets you tell a bad arguments fault from a broken tool fault. The tool output shows what the next step was reasoning over. Latency surfaces slow or timing out tools. Cost lets you attribute spend to a specific step.
Request-level traces are not enough for agents. A single failed top level call tells you the run failed, but not which of the twelve steps went wrong, nor whether the model or a tool was at fault. Step-level spans, linked in a tree, are what make a failed trajectory replayable instead of a guessing game.
Quick reference
TL;DR
Trace one span per agent step capturing the input context, the full model response, tool arguments, tool result or error, latency, and cost. Anything less makes failures unreproducible.
Imagine a detective trying to solve a case days after it happened. If nobody wrote down what each person said, what they did, and when, the detective is stuck guessing. An agent is the same. It takes many small steps to finish a task, and each step depends on the last. When something goes wrong at step twelve, you need a notebook entry for that exact step: what the agent was looking at, what it decided, what tool it reached for, what came back, how long it took, and how much it cost. If the notebook only has the final answer, you cannot replay the moment things broke. So good agent systems write a detailed diary entry for every single step, not just one summary at the end. That diary is what makes a confusing failure something you can actually sit down and debug.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
A throwaway prototype run locally where you can watch the console and inspect each step live. Full span instrumentation adds setup overhead that a single developer scratch loop does not need yet.
Spans would capture raw secrets, tokens, or regulated personal data into the trace store. Logging full context verbatim can violate privacy or compliance rules, so payloads need redaction first.
Extremely high throughput where storing every full prompt per step would dominate the cost budget. Verbatim per-step capture can cost more than the agent itself, so sampling or truncation is required.
A single shot LLM call with no loop and no tool use at all. There is only one step, so a plain request level log already captures the whole interaction.
Quick scan · 2–4 min
Everything important, quickly.
Open with why agents need step level spans rather than one request level trace, enumerate the six required fields, dwell on separating tool inputs from the model response and on the ephemeral reasoning trace, then close with OpenTelemetry GenAI conventions and tools like LangSmith, Langfuse, and Phoenix.
Real-world examples
Real products, models, and research that use this idea.
1LangSmith traces each LangChain or LangGraph step as a span with inputs, outputs, latency, and token cost, and renders the run as a clickable trace tree for replay.
2Langfuse is an open source agent observability platform that records nested spans per tool call and attributes cost per step across an entire trajectory.
3Arize Phoenix instruments agents using OpenTelemetry GenAI semantic conventions, so spans carry standardised attributes like model name, token counts, and tool call details.
4The OpenTelemetry GenAI semantic conventions define span attributes for LLM and tool calls, letting vendors like Datadog and Honeycomb ingest agent traces without a custom schema.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QHow do you keep trace payloads from leaking secrets or regulated data while still being useful for debugging?
Sign in or go Pro to see the approach for this follow-up.
QPer-step verbatim logging gets expensive at scale. How do you cut trace storage without losing the ability to debug failures?
Sign in or go Pro to see the approach for this follow-up.
QWhy are OpenTelemetry GenAI semantic conventions worth adopting instead of a homegrown span schema?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Logging only the final answer or one request level trace. Agents take many dependent steps, so you need a span per step or you cannot tell which step broke the run.