Context fidelity: maintaining and referencing information from earlier conversation turns. Single turn evals cannot measure this because each question is scored independently.
Imagine testing a waiter by asking one question at a time, rating each answer, and getting a high score. But when you have a full dinner conversation, the waiter forgets your allergy after you mentioned it, forgets your drink order, and asks for your name again even though you introduced yourself. The problem is not that the waiter gives bad individual answers. The problem is that the waiter does not remember what happened earlier. A single question test cannot catch this. You need a multi part conversation test where each question builds on the previous one.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
A chatbot that scores 4.2 out of 5 on single turn evaluation and gets user complaints about forgetting is showing a textbook eval gap. The eval measures one thing (individual response quality) and users experience another (conversation coherence). The missing dimension is context fidelity: the ability to maintain, reference, and build on information from earlier turns.
This gap is not a flaw in the single turn eval; it is a structural limitation. Single turn eval cannot measure what it does not test. Understanding why requires looking at how single turn eval works, what context fidelity actually requires, and how multi turn eval tests it.
Why single turn eval is blind to context fidelity
Single turn evaluation presents the model with a prompt, collects the response, and scores it. The next evaluation prompt is independent of the previous one. There is no conversation history, no accumulated state, no dependency between questions.
This means a model that generates excellent standalone responses scores identically to one that generates excellent responses while maintaining perfect conversation state. The eval cannot distinguish between them because it never tests the conversation dimension.
The 4.2/5 score in the scenario is real and valid for what it measures. Each individual response is genuinely good. The problem is that 'good individual responses' and 'good conversation' are different properties. A model can excel at one and fail at the other, and single turn eval only measures the first.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- MT-Bench uses two turn conversation pairs where the second turn depends on the first, specifically testing whether the model maintains context between turns.
- LMSYS Chatbot Arena captures multi turn conversations and rates them holistically, catching context fidelity failures that single turn benchmarks miss.
What an interviewer would ask next. Try answering before peeking at the approach.
QYour chatbot maintains context for 5 turns but loses it at turn 8. How would you diagnose whether this is a context window problem or an attention problem?
Check the token count at turn 8. If the conversation exceeds the context window, context loss is expected and the fix is truncation strategy or summarization. If the conversation is well within the window, the model is failing to attend to earlier turns. Test by placing the critical information at different positions (early, middle, late) in the context and measuring recall. This diagnoses whether the model has a positional attention bias.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Assuming a high single turn quality score guarantees good conversation experience, when single turn evals are blind to context tracking across turns.
60 second bullets to scan on the way to the call.
Why single turn eval cannot measure context fidelity.
How dependent question pairs test context tracking across turns.
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.