MT-Bench is used alongside Chatbot Arena. Describe what MT-Bench tests that single-turn benchmarks miss.

Flashcard·Easy·4.0 · 0·~30s

Attempt it

MT-Bench tests multi-turn conversation quality using 80 carefully crafted two-turn prompts across 8 categories: writing, roleplay, reasoning, math, coding, extraction, STEM, and humanities. A strong LLM judge scores each turn on a 1 to 10 scale. The critical insight is that MT-Bench catches models that handle the first turn well but fail on follow-up instructions. The second turn typically asks the model to revise, refine, or extend its first answer, testing context consistency and instruction-following ability that single-turn benchmarks never measure.

Your answer

Quick reference

TL;DR

MT-Bench uses 80 two-turn prompts across 8 categories scored by an LLM judge, catching models that fail on follow-up instructions and context consistency.

Memory aid

Easy to grasp

Imagine testing a tutor by asking one question and seeing if they give a good answer. Most tutors can do that. Now imagine asking a follow-up: 'Can you explain that differently?' or 'What if the numbers were larger?' A weak tutor falls apart because they cannot adjust their explanation or keep track of what they already said. MT-Bench tests AI models exactly this way. It asks a question, then asks a follow-up, and scores both turns. Models that look great on one-question tests sometimes stumble badly when the conversation keeps going.

Key concepts

MT-Bench Multi-Turn Evaluation LLM Judge Instruction Following

Detailed answer & concept explanation

~5 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Quick scan · 2–4 min

Everything important, quickly.

5 min: define MT-Bench as 80 two-turn prompts across 8 categories with LLM judge scoring, explain why the second turn tests skills that single-turn benchmarks miss, discuss LLM judge biases, mention the correlation with Arena Elo, and note limitations around the two-turn ceiling.

Real-world examples

Real products, models, and research that use this idea.

LMSYS publishes MT-Bench scores alongside Chatbot Arena Elo rankings, and the strong correlation between the two validates MT-Bench as a proxy for human preference.
Teams building customer support chatbots use MT-Bench-style evaluations to test whether their models handle follow-up questions without losing context.
Model development teams run MT-Bench after each training iteration as a quick signal for conversational quality, since it is much faster than collecting live Arena votes.
The 8-category breakdown allows teams to identify specific weaknesses: a model might score well on math and coding but poorly on roleplay and creative writing.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QHow do LLM judge biases (position, length, self-enhancement) affect MT-Bench scores, and how can they be mitigated?

QWhy does MT-Bench correlate with Chatbot Arena Elo, and where do the two evaluations diverge?

QHow would you extend MT-Bench to test conversations longer than two turns?

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Assuming MT-Bench is just another single-turn benchmark when its core value is the two-turn structure that tests follow-up handling and context consistency.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

References

Primary sources. Skim if you want the original framing.