A single-turn eval shows your chatbot scores 4.2/5. But users complain it 'forgets what they said.' Which eval dimension is your single-turn setup missing? · Zenaique

Attempt it

Quick reference

TL;DR

Context fidelity: maintaining and referencing information from earlier conversation turns. Single turn evals cannot measure this because each question is scored independently.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Imagine testing a waiter by asking one question at a time, rating each answer, and getting a high score. But when you have a full dinner conversation, the waiter forgets your allergy after you mentioned it, forgets your drink order, and asks for your name again even though you introduced yourself. The problem is not that the waiter gives bad individual answers. The problem is that the waiter does not remember what happened earlier. A single question test cannot catch this. You need a multi part conversation test where each question builds on the previous one.

Key concepts

Multi-turn Evaluation Context Fidelity

Detailed answer & concept explanation

~4 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Sign in for 5 free unlocks/day or go Pro for unlimited. See Pro pricing.

Quick scan · 2–4 min

Everything important, quickly.

Open by naming the missing dimension: context fidelity. Explain the structural blind spot of single turn eval, where each response is scored independently. Walk through dependent question pair design and how MT-Bench implements it. Cover the degradation curve: how context fidelity worsens with conversation length. Dismiss the three distractors by mapping each to a different symptom. Close on how to build multi turn eval at scale with synthetic conversation generation.

Real-world examples

Real products, models, and research that use this idea.

MT-Bench uses two turn conversation pairs where the second turn depends on the first, specifically testing whether the model maintains context between turns.
LMSYS Chatbot Arena captures multi turn conversations and rates them holistically, catching context fidelity failures that single turn benchmarks miss.
Customer support chatbots at companies like Zendesk and Intercom measure conversation resolution rate, which implicitly requires context fidelity because resolving an issue requires tracking the problem across multiple turns.
Anthropic and OpenAI both evaluate their models on multi turn conversation benchmarks internally, measuring context tracking degradation as conversation length increases.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QYour chatbot maintains context for 5 turns but loses it at turn 8. How would you diagnose whether this is a context window problem or an attention problem?

Sign in or go Pro to see the approach for this follow-up.

QHow would you build a multi turn eval that tests context fidelity at scale without manual conversation design?

Sign in or go Pro to see the approach for this follow-up.

QA model has perfect context fidelity for 10 turns but its single turn quality drops as the conversation gets longer. What is happening?

Sign in or go Pro to see the approach for this follow-up.

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Assuming a high single turn quality score guarantees good conversation experience, when single turn evals are blind to context tracking across turns.

Sign in or go Pro to see all red flags and common mistakes. See Pro pricing.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.