How do you evaluate a customer service chatbot that handles multi-turn conversations? What dimensions matter?
Measure task completion, per-turn relevance, context fidelity, tone, and escalation appropriateness. Handle branching with scripted scenario trees and per-turn LLM-as-judge scoring.
Imagine grading a phone call between a customer and a support agent. You would not just ask "was the call good?" at the end. You would listen to each part: did they understand the problem, did they remember what the customer said earlier, were they polite, and did they solve it? A chatbot eval does the same thing, checking each message in the conversation separately, because one terrible response in an otherwise good conversation is what makes customers angry.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Evaluating a multi-turn customer service chatbot is fundamentally different from evaluating a single-turn question-answering system. The conversation has state that evolves over turns, branches when users say unexpected things, and requires emotional calibration alongside factual accuracy.
The eval design has three pillars: the right dimensions to measure, a systematic approach to conversation branching, and per-turn scoring that catches single-turn failures invisible to aggregate metrics. Each pillar addresses a specific failure mode that production chatbots exhibit.
The five evaluation dimensions
Task completion rate is the outcome metric: did the bot resolve the customer's issue without requiring escalation to a human agent? This is the metric the business cares about most, but it does not explain why a conversation succeeded or failed.
The four process metrics fill that gap. Per-turn relevance checks whether each bot response addresses what the user just said, catching the common failure where the bot responds to a previous topic instead of the current one. Context fidelity checks whether the bot correctly references information from earlier turns, catching the "it forgot what I said" complaint. Tone and empathy check whether the bot's emotional register matches the situation. Escalation appropriateness checks whether the bot hands off to a human at the right moment.
All five dimensions are measurable with an LLM-as-judge rubric. The rubric converts subjective qualities like "appropriate empathy" into concrete scoring criteria: "If the customer mentions financial loss, the response must acknowledge the impact before offering a solution."
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Anthropic's Claude is used as an LLM-as-judge in enterprise customer service deployments, scoring per-turn relevance and tone against rubrics calibrated to the company's service standards.
- MT-Bench tests two-turn conversations with generic follow-ups, demonstrating the gap between benchmark multi-turn and production multi-turn evaluation.
What an interviewer would ask next. Try answering before peeking at the approach.
QYour scenario scripts cover 50 conversation types but production traffic includes a long tail of unusual requests. How do you evaluate the long tail?
Sample unusual conversations from production logs weekly, cluster them by failure mode, and add representative scenarios to the eval set. The long tail will never be fully scripted, so complement scripted eval with production monitoring: per-turn LLM-as-judge on a rolling 1% sample of live traffic.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Using a single end of conversation score that averages away a catastrophic turn-5 failure into an acceptable-looking aggregate metric.
60 second bullets to scan on the way to the call.
Name the five evaluation dimensions for a customer service chatbot
Explain why per-turn scoring catches failures that aggregate metrics hide
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.