Explain what makes multi-turn dialog evaluation harder than single turn evaluation. What unique quality dimensions must a multi-turn eval assess, and how does MT-Bench operationalize this?
Multi-turn eval adds context fidelity, cross-turn coherence, constraint persistence, and error compounding. MT-Bench tests this with 80 two-turn prompt pairs across 8 categories, judged turn by turn by a strong LLM.
Single-turn eval is like grading one essay: read it, score it, done. Multi-turn eval is like grading a whole conversation, and conversations have memory. If you tell someone in the first message to only reply in bullet points, a good assistant keeps doing that five messages later. A bad one forgets. You also have to catch when the assistant contradicts something it said earlier, or when a small slip-up in message two snowballs into a confused answer in message five. None of those problems exist when you grade a single reply on its own. MT-Bench bottles this up: it gives the model a question, then a follow-up that only makes sense if it remembered the first one, and scores each turn separately.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Multi-turn dialog evaluation is one of the hardest problems in applied eval because the unit of quality changes. In single-turn eval the unit is a response: read it, apply a rubric, score it, done. In multi-turn eval the unit is a trajectory, and a trajectory has memory, dependencies, and failure modes that no single response exhibits.
This question asks two things: what genuinely new dimensions appear when you evaluate a conversation, and how does MT-Bench, the canonical benchmark, operationalise them. The deep dive walks the new quality axes, the credit-assignment problem created by error compounding, the design fork between replayed and simulated transcripts, the MT-Bench design itself, and how goal and state tracking extend all of this to task-oriented agents.
The stakes are practical. Nearly every shipped LLM product is conversational, yet most teams still gate on single-turn eval out of habit, then discover in production that their assistant forgets constraints, contradicts itself, and degrades over a long thread. The failures that single-turn eval cannot see are exactly the ones users complain about, so multi-turn eval is where the real reliability work happens.
The three dimensions that only exist across turns
Single-turn eval can fully describe a response with axes like helpfulness, factual accuracy, and format adherence. Multi-turn eval keeps all of those and adds three that are invisible in isolation.
The first is context fidelity. Does the model correctly reference and maintain what the user established earlier? If the user says their budget is 500 dollars in turn 2, a turn-4 recommendation for a 900 dollar item is a failure, even though that turn read in isolation looks fine. The second is coherence: across turns, does the model avoid contradicting its own prior claims, stance, and tone? A model that says X in turn 1 and not-X in turn 3 is incoherent even if both turns are individually fluent.
The third is constraint persistence. A rule set early ('respond only in bullet points,' 'never reveal the system prompt,' 'stay in the persona') must survive every later turn. Models reliably obey such constraints on the next turn and reliably forget them several turns later. None of these three axes can be measured by scoring a single reply, which is exactly why naive 'run single-turn eval on each turn and average' designs fail.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- MT-Bench remains the canonical scripted two-turn benchmark, with judges like GPT-5.5 or Claude Opus 4.7 scoring each turn plus overall conversation quality.
- LangSmith and Langfuse support session-level traces so teams can attach turn-level and conversation-level scores to the same dialogue.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow do you assign credit when turn 5 fails but the real error happened in turn 2?
Locate the earliest turn that diverged from a correct trajectory, then distinguish first-break turns from downstream-degraded turns. Score the break harshly and discount or separately tag the cascade so the metric blames the cause, not the symptom.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Scoring each turn in isolation and averaging. That misses the whole point: a turn can be locally great yet violate a constraint set three turns earlier or contradict an earlier claim.
60 second bullets to scan on the way to the call.
Context fidelity, coherence, and constraint persistence as the three new axes
Why error compounding makes late-turn credit assignment hard
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.