Context fidelity (referencing and maintaining information from earlier turns) is unique to multi turn eval. A single turnprompt has no prior context to carry, so it cannot test carryover.
Think of evaluating a waiter. A single turntest is one customer who says everything in one breath: 'A coffee, no sugar, oat milk.' Easy to grade: did the order arrive right? A multi turntest is a real conversation: 'I'll have a coffee.' Then later, 'make it oat milk,' then later still, 'actually no sugar.' Now the hard question is whether the waiter remembers and combines all three turns into one correct order. That memory-across turns ability is context fidelity. The single-customer test literally cannot measure it, because nothing was said earlier to remember. Length, word choice, and speed matter in both cases, so they are not what makes the conversation harder to grade.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Multi-turn dialog evaluation asks a question single turneval structurally cannot: did the model carry the conversation correctly across turns? The right answer here is context fidelity, the model's ability to reference, maintain, and act on information established in earlier turns. Every other option describes something that is either present in single turntoo (response length), not a real quality dimension (vocabulary diversity), or a system metric rather than an output-quality axis (latency).
The deep reason context fidelity is unique is that single turnand multi turn eval grade different objects. Single-turn grades one response against one prompt. Multi-turn grades a trajectory of responses against an evolving conversation state. Fidelity to prior turns only has meaning once there are prior turns to be faithful to. This deep dive walks the failure taxonomy that fidelity covers, the two scoring lenses you need, how goal completion is measured across a session, and the measurement traps that catch teams who bolt single turn harnesses onto conversational logs.
Why context fidelity cannot exist in single turneval
In a single turnevaluation, each example is a self-contained prompt and a self-contained expected response. The model sees one message and replies. There is no earlier turn, so there is no prior context to remember, reference, or contradict. Whatever the model needs is in the prompt in front of it.
The instant a conversation has a turn 2, that changes. The model now has history, and history is something it can get wrong. It can honour an earlier constraint or quietly drop it. It can stay consistent with an earlier claim or contradict it. It can keep tracking the user's goal or lose the thread. None of these behaviours are even definable without prior turns.
That is the precise sense in which context fidelity is unique to multi turn. It is not merely harder in multi-turn; it is undefined in single turn. This is also why the distractors fail: length, vocabulary, and latency all exist identically whether or not there is conversation history, so none of them is the dimension that multi turnnewly introduces.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- MT-Bench scores two-turn conversations with an LLM judge, the canonical benchmark that surfaced multi turn context fidelity as a distinct axis.
- LangSmith and Langfuse support session-level traces so evaluators can score a full conversation, not just isolated turns.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you score a 10-turn conversation where turn 3 derails and every later turn inherits the bad context?
Separate the origin turn from propagation turns so one root failure is not counted ten times. Score per turn for localisation, then judge the session holistically for goal completion. Attribute the failure to turn 3, not the whole tail.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Reusing single turnmetrics on multi turndata. Scoring each turn in isolation misses context carryover and whether the conversation reached the user's goal across the whole session.
60 second bullets to scan on the way to the call.
Why context fidelity cannot exist in single turnevaluation
Per-turn scoring versus whole-session scoring and what each catches
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.