A single-turn eval shows your chatbot scores 4.2/5. But users complain it 'forgets what they said.' Which eval dimension is your single-turn setup missing?

The missing dimension is **context fidelity**: the ability to maintain, reference, and build on information from earlier turns in a conversation. Single turn evals score each response independently. They ask the model a question, rate the answer, and move to the next question. There is no memory between questions. This means a model can score 4.2/5 on every individual response while completely failing to track what the user said three turns ago. The user complaint about forgetting is the exact symptom of missing context fidelity. The model gives good answers to each question in isolation, but it does not connect them into a coherent conversation. Multi turn eval fixes this by constructing dependent question pairs. Turn 1: 'I am allergic to shellfish. Recommend a restaurant.' Turn 2: 'What should I order there?' If the model forgets the allergy, it might recommend shrimp. A single turn eval would never catch this because it would score turn 2 without the context of turn 1. MT-Bench includes multi turn pairs specifically to test context tracking. Custom golden sets extend this to longer conversations with explicit callbacks to earlier information.

The missing dimension is **context fidelity**: maintaining, referencing, and building on information from earlier conversation turns. Single turn evals score each response independently, so they are structurally blind to whether the model tracks conversation state. A model can score 4.2/5 on individual responses while losing track of user preferences, prior answers, and conversational context after two or three turns. The symptom users describe as 'forgetting' is exactly this gap. Multi turn eval tests context fidelity by constructing scenarios where later questions depend on earlier answers. MT-Bench includes dependent turn pairs for this purpose. Custom multi turn golden sets push further by chaining 5 to 10 turns with explicit callbacks ('earlier you said X; does that still apply?'). The eval scores whether the model incorporates prior context, not just whether each individual response is high quality. Factual accuracy, response diversity, and instruction following are all real eval dimensions, but none explain the specific symptom of forgetting prior context. Context fidelity is the dimension that maps directly to the user complaint.