Tag every speaker turn with a structured speaker.name attribute and give the selector its own span. Aggregations group by speaker.name.
Imagine a panel show with three guests, a host, and a single microphone. Each time someone speaks, you tape a name tag onto that recording so later you can replay only the critic's comments, or only the researcher's. The host's decision about who to call on next is also recorded, as its own little clip, so you can review why the critic got the mic three times in a row. Without those name tags, the tape is one long unlabeled blob and you can never answer who talked the most or who said the most useful things.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Multi-agent GroupChats are the single hardest shape to trace cleanly in modern LLM systems. You have multiple speakers taking turns, a selector picking who speaks next, each turn potentially emitting multiple LLM calls and tool calls, and a shared message history threading the whole thing together. The naive instrumentation, just letting the LLM SDK emit one span per call, gives you a flat list of model invocations with no way to ask 'how much does the critic actually cost' or 'is my selector picking the executor too often'.
The fix is a three-span shape per round plus two structured attributes. The shape encodes the orchestration tree so dashboards can group by speaker, by selector kind, or by round index. The attributes carry the identity information without inflating the name index. Both pieces matter: a clean tree without attributes does not answer per-speaker questions, and attributes without the tree do not answer per-round or per-policy questions.
Mental model: the trace tree mirrors the orchestration tree. One selector span per round, one step span per speaker turn, one LLM or tool span per model or function call. Identity is an attribute, not a name.
The three-span shape per round
Every round of a GroupChat decomposes into three logical events. Trace each separately.
Selector span
The selector is whatever component decides who speaks next. It might be a rule (round-robin), an LLM call (let GPT decide), or a learned policy. Emit a selector span with attributes:
- selector.kind = 'round-robin' or 'llm' or 'rule-based'
- selector.picked = 'critic' (the speaker chosen)
- selector.reason (optional, short string)
If the selector is itself an LLM call, the selector span has one child LLM span carrying standard gen_ai attributes. This is how you make selector cost visible separately from speaker cost.
Speaker step span
One step span per speaker turn. Attributes:
- speaker.name = 'researcher' (or 'critic', or 'executor')
- speaker.role = the broader category (analyst, reviewer, executor)
- turn.index = monotonic round counter
All LLM calls and tool calls the speaker emits during this turn become children of this step. They inherit the speaker attribute by tree position; do not repeat it on every child.
LLM and tool spans
Standard gen_ai attributes only. model, input_tokens, output_tokens, finish_reason, temperature. Tool spans get a tool.name and tool.kind. Speaker identity stays on the parent step.
This shape gives you exactly the queries you need: group by speaker (sum costs, count refusals), group by selector kind (compare policies), group by round index (find blow-up rounds).
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- AutoGen 0.4 emits per-speaker spans through its OpenTelemetry integration, including a selector span for the routing decision.
- LangGraph multi-agent workflows use checkpoint ids as the equivalent routing-visibility primitive, captured by LangSmith automatically.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you trace a GroupChat where the selector is itself an LLM call?
The selector span becomes a span with a child LLM span using standard gen_ai attributes plus selector.kind = 'llm'. Costs of the selector are then visible separately from the speakers, and you can A/B against rule-based selectors.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Flattening every speaker's LLM call into the same generic span shape. The trace becomes a list of model calls with no way to filter by speaker, and per-speaker cost or refusal questions become unanswerable.
60 second bullets to scan on the way to the call.
Why speaker identity belongs in a structured attribute, not in the span name
What the selector span captures and why it is separate
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.