Design an observability and debugging system for a production multi-agent application where 5+ agents communicate, share state, and collectively complete tasks. What do you trace, how do you correlate, and what failure modes specifically does it have to surface? Include the trace shape, the stack choice, and the visualisation requirements that make a 50-message multi-agent run actually debuggable.
Free · 2 AI evals / day
A production multi-agent system with five or more agents is far harder to debug than a single agent. Each agent has its own loop, its own state, its own decisions, and the agents talk to each other. When the system fails, the failure is in the **interaction**, not in any one agent in isolation.
The foundation is a **shared task-correlation id**. Every agent and every tool call inside one task carries the same id. Without it, you cannot tell which agent's message belongs to which task, especially when the system runs many tasks concurrently. The id is the join key for the entire trace.
For each task you trace **three kinds of events**. Inter-agent messages, where one agent sends something to another, including the sender's reasoning and the message payload. Per-agent metrics, like token cost, latency per step, and how many steps each agent took. Per-tool spans, with arguments and results.
The failure modes you need to surface are specific to multi-agent systems. **Deadlocks** are two agents waiting for each other. **Cascading errors** are when one agent's bad output poisons the next. **Runaway delegation** is when an agent keeps spawning sub-agents without converging. **Role drift** is when an agent stops following its persona.
Text logs of a 50-message multi-agent run are unreadable. Nobody scrolls 2000 lines looking for a deadlock. The visualisation is a graph: agents are nodes, messages are edges, time is the layout axis. LangSmith and Langfuse both ship multi-agent topology views in 2026.
Three pillars.
**Correlation and identity on every span.** A shared task-correlation id propagates across every agent and every tool call. Every span also carries the agent identity (which agent emitted it). Together they let you query 'show me everything agent C did inside task T' or 'show me the full inter-agent message flow for task T'. Without correlation, the trace is a soup; without identity, attribution after handoff is impossible.
**Three event types.** Per-message records capture every inter-agent message: sender agent, recipient agent, message type, payload, and the sender's reasoning that produced it. Per-agent metrics capture token cost, latency per step, step count, tool calls, errors, and decisions made. Per-tool spans capture name, arguments, result, and duration. The per-message records are central because the system's behaviour lives in the message flow, not in any one agent's internal state.
**Named failure-mode detectors.** Multi-agent systems fail in specific named ways the observability has to surface: deadlocks (two agents blocked on each other), cascading errors (one agent's bad output propagates), runaway delegation (recursive sub-agent spawning without convergence), role drift (an agent overlapping another agent's responsibility), and broken handoffs (messages consumed by the wrong agent). Each gets a detector that runs over the trace and emits an alert.
**Stack choice.** LangGraph plus LangSmith on the Anthropic stack, or OpenTelemetry plus Langfuse for vendor-neutral, with a topology view on top.
**Visualisation is non-negotiable.** A graph of agents as nodes and messages as edges, time-ordered, with token cost and latency overlaid. Replay from checkpoints lets engineers reproduce a specific failure, which matters because multi-agent runs are non-deterministic.
Quick reference
TL;DR
Shared task-correlation id on every span, per-message and per-agent tracing, named failure-mode detectors, and a graph topology view because text logs of a 50-message run are unreadable.
Imagine watching five chefs in a kitchen all making one big meal together. If you only listen to one chef at a time, you cannot tell why the meal came out wrong. You need a map of the kitchen showing which chef handed what to which other chef, in what order, how long each step took, and what each chef was thinking when they did it. Every dish has a sticker on it that says which order it belongs to, so you can follow one order across the whole kitchen. When the meal goes wrong, you look at the map for that order's sticker, and you can see exactly where one chef passed something burnt to the next chef and where the dish went sideways. A multi-agent observability system is the kitchen map.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
A single-agent system with no inter-agent communication at all. The multi-agent failure modes (deadlock, role drift, runaway delegation) do not apply; single-agent observability with standard LLM tracing is sufficient.
A purely batch pipeline with deterministic agent execution and no concurrency. Replay-from-input works directly, so the heavy time travel from checkpoint machinery is overhead without benefit.
A development prototype where the cost of building topology visualisation outweighs the debug value. In prototype work, text logs plus a small number of agents is often workable; the investment in graph visualisation pays off at production scale.
Quick scan · 2–4 min
Everything important, quickly.
Open with the framing shift: the unit of observability moves from the agent to the conversation graph. Name the shared task-correlation id and the agent-identity tag as the two non-negotiable spans of metadata. List the three event types: inter-agent messages with reasoning, per-agent metrics, per-tool spans. Walk the four canonical failure modes (deadlock, cascade, runaway, drift) with a detector for each. Pick LangGraph plus LangSmith or OpenTelemetry plus Langfuse as the stack. Justify why text logs are unreadable and the graph topology view is non-negotiable. Close on replay, sampling, retention, and async emission as production disciplines.
Real-world examples
Real products, models, and research that use this idea.
1LangSmith ships a multi-agent topology view that visualises agents as nodes and inter-agent messages as edges, with token cost and latency overlaid; this is the canonical reference for the visualisation requirement.
2Langfuse provides vendor-neutral multi-agent tracing with OpenTelemetry compatibility, used by teams that want to avoid LangChain stack lock-in.
3LangGraph's checkpointer enables time-travel replay of multi-agent runs from any prior checkpoint, which is how non-deterministic failures get reproduced.
4Anthropic's research on building effective agents explicitly calls out per-message reasoning capture and shared correlation as preconditions for debugging multi-agent systems.
5OpenTelemetry's GenAI conventions added agent-identity and inter-agent-message span attributes in 2026, formalising the multi-agent trace shape across vendors.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QHow do you detect role drift specifically, given that the LLM is the one emitting the messages and judging the messages?
Sign in or go Pro to see the approach for this follow-up.
QMulti-agent runs are non-deterministic. What does replay actually require to be useful?
Sign in or go Pro to see the approach for this follow-up.
QHow do you make the observability cost itself not become a production bottleneck on a 50-message run?
Sign in or go Pro to see the approach for this follow-up.
QHow does the design change when agents communicate asynchronously via a shared store rather than direct messages?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Treating multi-agent observability as a stack of single-agent traces. Without a shared correlation id and a graph topology view, a 50-message run is unreconstructable from text logs alone.