Which events must every agent execution trace capture to enable debugging of production failures?
A useful agent trace records per step: input context, the model output with its tool call, the tool inputs and result or error, plus latency and token cost.
Imagine you hire an assistant to plan a trip, and they make twenty phone calls to book flights and hotels. Later the trip falls apart. To find out what went wrong, you need a recording of each call: who they called, exactly what they asked, what answer they got, and how long each call took. If all you have is the final ruined itinerary, you cannot tell which call caused the mess. An agent trace is that recording. For every step the agent takes, it saves what the agent saw, what it decided to do, which tool it called and with what inputs, what the tool returned, and the time and money that step burned. With that recording you can replay the whole run and find the exact step where things broke.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
An agent trace is the structured, per-step record of everything that happened during one agent run. An agent is an LLM in a loop with tools, so a single run can span dozens of model calls and tool calls. When that run fails, and non-deterministic multi-step systems fail constantly, the only way to find the broken step is to have recorded each step as it happened.
The question this answer settles is precisely which events a trace must capture. The right list is small and principled: the input context, the model response with its tool call, the tool inputs, the tool output or error, the step latency, and the step token cost. Everything else is either noise or run-level aggregate metadata. The discipline is to capture exactly what you would need to replay and explain the run, and to resist logging data that never helps root-cause a failure.
The stakes are higher than for ordinary software. A traditional service either throws an exception or returns the right value, and a stack trace points at the line that broke. An agent fails silently far more often. It returns a confident, well-formatted answer that happens to be wrong, with no exception anywhere in the stack. The trace is the only artefact that exposes that class of failure, which is why observability is treated as a hard requirement for production agents rather than a nice to have.
Why per-step capture is the whole point
A plain LLM call is easy to debug. There is one input and one output. An agent is different. It loops, and the reasoning the model produces between turns is ephemeral. Once the turn ends and the next turn begins, the chain of thought that led to a tool call is gone unless you saved it. State also mutates on every iteration, so the prompt the model sees on step twelve is not the prompt it saw on step one. You cannot reconstruct step twelve's context from the final state alone.
This is why the final answer alone is useless for debugging. Suppose a twenty-step research agent returns a confidently wrong summary. The final answer tells you the run failed. It does not tell you that on step seven the model called a search tool with a malformed query, got an empty result, and then hallucinated a plausible answer to cover the gap. The failure is real, but it is buried thirteen steps before the output you can actually see.
There is also a compounding-error angle. Per-step error rates multiply across a long run, so a tool that succeeds ninety percent of the time still fails roughly a third of the time over ten steps. When the end to end success rate is governed by dozens of fallible steps, you cannot improve reliability without knowing which step is the weak link. Only per-step capture surfaces that.
To recover that story you need a record of each step in sequence. The trace turns an opaque black box into a replayable transcript. That replayability is the single most important property of an agent observability system, because replay is what converts a vague report of a bad run into a precise, reproducible bug.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
| Field | Capture? | Why |
|---|---|---|
| Input context snapshot | Yes | Shows what the model saw that turn |
| Model response plus tool call | Yes | Shows what the model decided |
| Tool inputs and output or error | Yes | Shows what the tool actually did |
| Step latency and token cost | Yes | Enables performance and cost attribution |
| Operating system and browser | No | Irrelevant to agent reasoning |
| Sampling seed | No | Rarely exposed by the API, not a standard field |
Real products, models, and research that use this idea.
- LangSmith renders each agent run as a nested tree of spans, one per LLM call and tool call, with token cost and latency on every node, so you can click into the exact failing step.
- Langfuse models a run as a trace containing observations for each generation and tool call, capturing inputs, outputs, model, and cost for replay and evaluation.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you keep traces useful when a single tool returns a 50 kilobyte response on every one of forty steps?
Store the full payload in a side store keyed by an id and put only a truncated preview plus the id in the span. Reference the full blob on demand so the trace stays browsable without losing replay fidelity.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Logging only the final answer. When a 20-step run fails, the final output tells you nothing about which step broke. You need a per-step span with inputs, outputs, and tool results.
60 second bullets to scan on the way to the call.
Name the four core things a step span must record.
Explain why the final answer alone is insufficient for debugging.
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.