Select every metric that belongs on a serious multi-agent eval dashboard
Five real metrics (success, per-agent step correctness, tool precision, cost and latency, branch coverage) and one vanity measure (prompt length) that does not move with quality.
Imagine grading a relay race team. You care about whether they finished, whether each runner ran their leg correctly, whether the baton handoffs were clean, how long the whole race took, and whether you actually tested every leg of the course. You do not care how many words their coach wrote on their warm-up sheet. The warm-up sheet might be helpful or useless, but its character count tells you nothing about the race. The eval dashboard for a multi-agent system follows the same logic. Measure what affects the outcome; ignore what does not.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Evaluating a multi-agent system is not single-agent evaluation with more rows in the spreadsheet. The output you are scoring is a trajectory, not a single answer, and the failure modes (loops, role bleed, misroutes, cost explosions) only show up when you measure the right things in the right granularity. A serious dashboard reflects that.
This question separates engineers who have actually shipped multi-agent in production from those who copied a metrics list from a blog post. The trap option is deliberately the most measurable one, which is the lesson: measurability is not the same as importance.
One-line summary: the dashboard must answer 'did it work, who broke it, were the tools right, what did it cost, did we test every path'. Five questions, five metrics. Everything else is noise.
The five real metrics, what they catch, and what they miss
End to end task success
The non-negotiable headline. You hold a golden set of representative tasks with expected outputs and a scoring function (exact match where possible, rubric or LLM judge where unavoidable). End to end success is the only metric whose movement tells you the user-visible system is working or not.
What it misses: it does not tell you why a failure happened. It does not separate cost regressions from quality regressions. You need the other four to localise the signal.
Per-agent step correctness with attribution
The debugging surface. Each agent's output is scored against a per-step ground truth or per-step rubric. The attribution piece (which agent produced the first wrong intermediate) is what makes this actionable. Without it, you know something broke; with it, you know what to fix.
Failure mode this catches: end to end success drops 5 points after a model upgrade, per-agent metrics show the planner is unchanged but the critic is 8 points worse, you immediately know to retune the critic prompt or roll back the critic's model.
Tool-call precision
For every tool call in a trace, check the tool name and argument shape. Two things go wrong: the model picks the wrong tool (a routing error) or the model picks the right tool with malformed args (a JSON regression). Both are silent in end to end metrics because the agent often recovers by retrying.
Failure mode this catches: a model update changes how tool-use JSON is emitted, your retry logic absorbs it, end to end success drops only 2 points but tool-call precision drops 25 points. That precision drop is the smoking gun.
Cost and latency per task
Total tokens by agent, dollars per run, wall-clock p50 and p95. In 2026 these are quality metrics, not engineering metrics. A 10 percent accuracy gain at 5x cost is a regression. A 100ms latency gain that lifts task success by 2 points is a real win.
Branch coverage on the topology graph
The multi-agent analogue of code coverage. For each edge in the topology (supervisor to coder, supervisor to critic, critic to coder, etc.), what fraction was exercised by the eval set? Untested branches will break first in production. Weight by expected traffic when prioritising.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- LangSmith dashboards expose end to end success, per-step traces, tool-call inspection, token-cost and latency by run.
- Anthropic's published agent guidance recommends per-step correctness with attribution as the primary debugging metric.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you actually attribute a failure to a specific agent in a trace?
Tag every message with (turn, source_agent, parent_message_id) at runtime, compare the agent's output against per-step ground truth or a rubric, and walk the dependency chain from the final wrong answer backward to the first wrong intermediate. The agent that produced the first wrong intermediate is the attributed agent.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Picking system prompt length because it is easy to measure. Easy to measure plus uncorrelated with outcomes is the definition of a vanity metric.
60 second bullets to scan on the way to the call.
Why is end to end success on a golden set the headline metric for a multi-agent system?
Why does per agent step correctness require attribution to be useful for debugging?
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.