What makes multi-turn dialog evaluation harder than single-turn, and how does MT-Bench address it?

**Multi-turn dialog evaluation** asks whether a model holds up across a whole conversation, not just on one reply. Single-turn eval reads one response and scores it in isolation. That misses everything that only shows up over time. Three dimensions are unique to multi-turn. First, context fidelity: does the model remember what the user said earlier and reference it correctly? Forgetting a detail from turn 2 when answering turn 4 is invisible to single-turn eval. Second, coherence: does the model stay consistent, or does it contradict its own earlier position, tone, or facts? Third, constraint persistence: if turn 1 says 'reply only in bullet points,' does the model still obey that by turn 5? There is also error compounding. A small mistake in turn 2 that nobody corrects can cascade into worse answers in turns 3 through 5. The conversation degrades even if each later turn looks reasonable on its own. MT-Bench operationalises this with 80 two-turn prompt pairs across 8 categories like writing, reasoning, math, and coding. The follow-up question needs context from the first turn, and a strong LLM judges each turn separately plus the overall quality.

**Multi-turn evaluation** is harder because quality stops being a property of one response and becomes a property of the whole trajectory. Single-turn eval asks: was this reply good alone? Multi-turn must also ask whether it fits everything before it. Three dimensions only exist across turns. Context fidelity means the model correctly references earlier user statements. Coherence means it never contradicts its own prior positions, facts, or tone. Constraint persistence means a rule set in turn 1 must survive to turn 5. Error compounding is the deepest issue. A slip in turn 2 cascades, so a late failure may not be the late turn's fault. That is a credit-assignment problem absent from single-turn scoring. MT-Bench operationalises this concretely. It uses 80 two-turn prompt pairs across 8 categories: writing, roleplay, reasoning, math, coding, extraction, STEM, and humanities. The second turn requires context from the first, so a model that drops context visibly fails. A strong LLM judge scores each turn separately and the conversation overall. Crucially, the second-turn score is conditioned on the first turn being in context. This is what makes MT-Bench a multi-turn benchmark, not two single-turn ones.