You are building the eval set for a 5 agent customer support team (router, billing, returns, tech support, escalation). Design a golden set that exercises the branching paths and the failure modes, not just the happy path.
Cover the decision graph, not the demo path: enumerate first-hop and handoff edges, slice into happy/ambiguous/escalation plus adversarial, attach both route-gold and output-gold to every example, and gate cost per task.
If you are testing a phone-tree at a bank, press 1 for accounts, 2 for cards, 3 for loans, you would not test only the 'press 1 then resolve' path. You would call with a question that sounds like accounts but is really about cards, to see if the human transfers you correctly. You would call with an angry tone to see who picks up. You would call in another language. A golden set for a 5-agent support team is the same idea: write down the question, the right specialist who should answer, the path the call should follow, and how much it should cost. Then check all of it: not just the final answer, but who handled it and how many hops it took.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Designing a golden set for a multi-agent system is fundamentally different from designing one for a single-agent system. The new failure modes are about the path: wrong specialist chosen, infinite hand-offs, escalation missed, cost compounding across hops. A single agent style eval that only checks final-output correctness will report green while every one of these failures ships to production.
The right framing is graph coverage. The system is a router plus specialists graph; the golden set's job is to exercise every routing decision and every realistic handoff edge. Once you accept that framing, the design unfolds cleanly: enumerate the graph, slice by intent class to match real-world traffic and failure modes, attach multi-dimensional gold to every example, and gate on cost as well as quality.
This section walks through each step in detail, then through the operational discipline of maintaining the set over time as business logic and the model under test both change.
Graph coverage as the design primitive
Start with the picture. A 5-agent customer-support team is a directed graph: the router on top, four specialists below (billing, returns, tech-support, escalation), and edges showing realistic transitions.
First-hop edges (router to specialists): 4 edges, one per specialist. Every first-hop has to be exercised.
Handoff edges (specialist to specialist): Real business logic determines these. Common ones: returns → billing (refund processing), tech-support → escalation (capability gap), billing → escalation (account dispute), tech-support → billing (incident credit). Total: ~8-12 edges depending on policy.
Self-loops / no-handoff: every specialist sometimes resolves directly. These count as their own implicit edge, the 'no handoff' decision.
For coverage, each edge needs at least 3-5 examples to detect intermittent routing failures. With ~15 distinct edges, that pushes the golden set toward 50-75 examples just for edge coverage, before you add the slicing structure. Realistic golden sets for production support teams in 2026 run 150-300 examples.
Why per-edge coverage matters. A common bug pattern in multi-agent systems is that the router handles 4 of the 5 first-hops well and silently misroutes the 5th. Aggregate route-accuracy metrics that mix all examples together hide this, the misrouted specialist gets only 20 percent of traffic, so its errors are diluted in the average. Per-edge breakdowns surface the bad edge immediately.
The practical recipe: build a matrix with edges on rows and examples on columns; every cell that should be exercised gets at least three examples. The matrix is also the visual sanity check before launch, empty rows are coverage gaps.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Klarna's customer-support agent rollout in 2024 used a tiered eval with route-gold and per-agent attribution metrics, which surfaced router miscoverage before public launch.
- Intercom's Fin agent eval framework in 2026 separates intent-classification accuracy from resolution-quality scoring, mirroring the route vs final-output split.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you generate the ambiguous-routing slice without just making them up?
Sample real production tickets from a period where the routing system was making errors (or where humans were re-routing). Anonymize and use those as the ambiguous slice. The cases where the system or humans struggled are exactly the ambiguous cases by definition. Synthetic generation tends to produce caricatures that the model handles easily.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Building the golden set out of resolved happy-path tickets only. The eval looks green because the easy cases pass, while routing errors and escalation failures stay invisible until production complaints surface them.
60 second bullets to scan on the way to the call.
Why graph coverage is the right framing for multi-agent eval design
How to enumerate first-hop and handoff edges before writing examples
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.