Design an observability and debugging system for a production multi-agent application where 5+ agents communicate, share state, and collectively complete tasks. What do you trace, how do you correlate, and what failure modes specifically does it have to surface? Include the trace shape, the stack choice, and the visualisation requirements that make a 50 message multi-agent run actually debuggable.
Shared task-correlation id on every span, per-message and per-agent tracing, named failure-mode detectors, and a graph topology view because text logs of a 50-message run are unreadable.
Imagine watching five chefs in a kitchen all making one big meal together. If you only listen to one chef at a time, you cannot tell why the meal came out wrong. You need a map of the kitchen showing which chef handed what to which other chef, in what order, how long each step took, and what each chef was thinking when they did it. Every dish has a sticker on it that says which order it belongs to, so you can follow one order across the whole kitchen. When the meal goes wrong, you look at the map for that order's sticker, and you can see exactly where one chef passed something burnt to the next chef and where the dish went sideways. A multi-agent observability system is the kitchen map.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
A production multi-agent system with 5 or more agents has a debugging surface fundamentally different from a single agent. The agents share state, hand off tasks, spawn sub-agents, and the failures live in the interactions rather than in any one agent's behaviour. An observability system that simply traces each agent separately is missing the structure that matters.
The correct design has three load-bearing pieces. A shared task-correlation id propagated across every agent and every tool call, plus agent identity on every span, gives you the addressability primitive. Three event types (per-message records with reasoning, per-agent metrics, per-tool spans) capture the data. Named failure-mode detectors turn the trace into an actionable signal. On top of all of this, a graph topology view is non-negotiable because text logs of a 50-message multi-agent run are unreadable.
The rest of this explanation walks the data model, the failure-mode detectors, the stack choices in 2026, the visualisation requirements, and the production discipline that makes multi-agent observability work at scale rather than just on paper.
The data model: correlation, identity, and three event types
Every multi-agent trace rests on two pieces of metadata that have to be on every span unconditionally. The shared task-correlation id is a single id assigned at task creation and propagated across every agent's calls and every tool call within the task. Without it, a multi-tenant production system with hundreds of tasks per minute produces a soup of spans that cannot be stitched into runs.
The agent identity is the second non-negotiable. Every span carries the identity of the agent that produced it. After the first inter-agent handoff, you cannot tell from message content alone which agent emitted what. Identity tagging is what makes attribution survive handoffs. Together, correlation id and agent identity let you query 'show me everything agent C did inside task T', which is the primitive every debug session starts from.
Three event types fill in the body. Per-message records capture every inter-agent message: sender, recipient, message type, payload, and (critically) the sender's reasoning that produced the message. The reasoning is what lets you debug an interaction by reconstructing what each agent thought it was doing. Most teams miss this when they first build observability; they capture the message but not the why, and end up unable to explain why an interaction went wrong.
Per-agent metrics capture token cost, latency per step, step count, tool calls, and decisions. These are how you find the bottleneck agent (one agent dominating runtime), the expensive agent (one agent burning the budget), or the stuck agent (one agent not making progress). The metrics aggregate across a task and across a fleet, feeding dashboards that show health at the system level.
Per-tool spans capture name, arguments, result, duration, and the agent identity of the caller. A tool span is just like a single-agent tool span with one extra field, but the extra field (identity) is what makes the data useful in the multi-agent context. A failing tool call has to be attributable to one agent; otherwise you cannot tell which agent's logic is responsible for the bad invocation.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- LangSmith ships a multi-agent topology view that visualises agents as nodes and inter-agent messages as edges, with token cost and latency overlaid; this is the canonical reference for the visualisation requirement.
- Langfuse provides vendor-neutral multi-agent tracing with OpenTelemetry compatibility, used by teams that want to avoid LangChain stack lock-in.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow do you detect role drift specifically, given that the LLM is the one emitting the messages and judging the messages?
Two paths. A static one: each agent has a declared role specification and an LLM judge tags every outbound message against the spec, with drift flagged when the tag deviates. A dynamic one: cluster outbound messages per agent over a run and flag when an agent's cluster overlaps another agent's cluster. Calibrate the judge against a small human-labelled set periodically to keep it from drifting too. The detector is itself an LLM judgement, so it inherits the calibration discipline.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Treating multi-agent observability as a stack of single-agent traces. Without a shared correlation id and a graph topology view, a 50-message run is unreconstructable from text logs alone.
60 second bullets to scan on the way to the call.
Name the shared task-correlation id and what it joins across.
Identify agent identity as the second non-negotiable tag on every span.
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.