What events must be logged in every agent execution trace for debugging to be tractable?
List the minimum set of fields that every agent execution span should capture. Explain why omitting any one of them makes production debugging significantly harder.
Trace one span per agent step capturing the input context, the full model response, tool arguments, tool result or error, latency, and cost. Anything less makes failures unreproducible.
Imagine a detective trying to solve a case days after it happened. If nobody wrote down what each person said, what they did, and when, the detective is stuck guessing. An agent is the same. It takes many small steps to finish a task, and each step depends on the last. When something goes wrong at step twelve, you need a notebook entry for that exact step: what the agent was looking at, what it decided, what tool it reached for, what came back, how long it took, and how much it cost. If the notebook only has the final answer, you cannot replay the moment things broke. So good agent systems write a detailed diary entry for every single step, not just one summary at the end. That diary is what makes a confusing failure something you can actually sit down and debug.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
An agent run is not one model call. It is a loop of dependent steps where each step's output becomes the next step's input. That structure dictates how you observe it. Observability for agents means capturing one trace span per step, not a single request level log for the whole run.
The reason is brutal and specific. When a twenty-step run produces a wrong answer, the bug lives in one step, and the model's reasoning that produced that step is gone the instant the turn ends. If the span did not record it, you cannot reproduce it, and you are left guessing. A request level trace tells you the run failed. It does not tell you which step, or whether the model or a tool was at fault.
This question is really asking for the minimum span schema that makes a failed trajectory replayable. The answer is six fields per step: the input context snapshot, the full model response including its reasoning, the tool arguments, the tool output or error, the step latency, and the step cost. The rest of this explanation justifies each field by the class of bug it lets you catch, then connects the schema to the OpenTelemetry GenAI conventions the industry is standardising on.
Why agents need step level traces, not request level
A traditional web service handles a request and returns a response, so one trace per request is enough. An agent is different. It takes many internal steps, each an LLM call plus a tool call, and each step's result steers the next. The failure mode is rarely the whole run. It is one step where the model misread an observation or a tool returned something unexpected.
If your trace granularity is the whole run, you can see that the final answer was wrong, but you cannot see where the trajectory diverged from a good path. You cannot tell whether step three fetched stale data, whether step seven hallucinated an argument, or whether step eleven misread a tool error.
Step-level spans fix this. Each span is one loop iteration, linked to its parent run, forming a trace tree. You can open the run, walk to the exact step that broke, and inspect everything that step saw and did. This is what people mean when they say agent debugging requires replay.
There is a second reason granularity matters: error rates compound. If each step succeeds ninety percent of the time, a ten-step run succeeds only about thirty-five percent of the time, and the failures cluster differently from run to run. Aggregate metrics over runs hide this entirely. You see a low end to end success rate but no signal about which step is the weak link. Step-level spans let you compute a per-step success rate, find the one tool that fails forty percent of the time, and fix the actual bottleneck instead of guessing.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- LangSmith traces each LangChain or LangGraph step as a span with inputs, outputs, latency, and token cost, and renders the run as a clickable trace tree for replay.
- Langfuse is an open source agent observability platform that records nested spans per tool call and attributes cost per step across an entire trajectory.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow do you keep trace payloads from leaking secrets or regulated data while still being useful for debugging?
Redact or hash sensitive fields at the instrumentation layer before export, keep a reversible reference id for authorised replay, and apply field level allow lists so prompts and tool args are scrubbed rather than dropped wholesale.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Logging only the final answer or one request level trace. Agents take many dependent steps, so you need a span per step or you cannot tell which step broke the run.
60 second bullets to scan on the way to the call.
Name the six fields every agent span should capture.
Explain why step level spans beat one request level trace for agents.
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.