Context fidelity (referencing and maintaining information from earlier turns) is unique to multi turn eval. A single turnprompt has no prior context to carry, so it cannot test carryover.
Think of evaluating a waiter. A single turntest is one customer who says everything in one breath: 'A coffee, no sugar, oat milk.' Easy to grade: did the order arrive right? A multi turntest is a real conversation: 'I'll have a coffee.' Then later, 'make it oat milk,' then later still, 'actually no sugar.' Now the hard question is whether the waiter remembers and combines all three turns into one correct order. That memory-across turns ability is context fidelity. The single-customer test literally cannot measure it, because nothing was said earlier to remember. Length, word choice, and speed matter in both cases, so they are not what makes the conversation harder to grade.
Detailed answer & concept explanation~8 min readEverything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Everything important, quickly.
4 min: why context fidelity is unique to multi turn, the carryover failure taxonomy, per turn vs whole-session scoring, goal completion, and why naive per turn averaging double-counts an early derailment.
Real products, models, and research that use this idea.
- MT-Bench scores two-turn conversations with an LLM judge, the canonical benchmark that surfaced multi turn context fidelity as a distinct axis.
- LangSmith and Langfuse support session-level traces so evaluators can score a full conversation, not just isolated turns.
- Anthropic and OpenAI red-team multi turn jailbreaks where a constraint stated early is eroded over later turns, a pure context-fidelity failure.
- Agent benchmarks like SWE-bench and GAIA score whole-trajectory goal completion rather than per-step replies in isolation.
- RAGAS and DeepEval ship conversation-level metrics distinct from their single turn faithfulness and relevance scores.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you score a 10-turn conversation where turn 3 derails and every later turn inherits the bad context?
QWhat does an LLM judge need to see to evaluate context fidelity, and what biases does that introduce?
Don't say thisRed flags and common mistakes that signal junior thinking. Click to expand.
Red flags and common mistakes that signal junior thinking. Click to expand.
Reusing single turnmetrics on multi turndata. Scoring each turn in isolation misses context carryover and whether the conversation reached the user's goal across the whole session.
The night-before-the-interview bullets. Scan these on the way to the call.
Primary sources. Skim if you want the original framing.
Same topic, related formats. Practice these next.