A colleague presents this claim: 'Our chatbot improved because BLEU score went from 0.32 to 0.41 on our eval set of open-ended customer questions.' Spot the flaw in this reasoning.
Click any words you think contain an error. Click again to unmark.
BLEU measures n-gram overlap and was designed for translation. On open-ended questions, excellent answers share few n-grams with references. A BLEU increase may reflect reference parroting, not genuine improvement.
Imagine grading an essay by counting how many exact word sequences match the teacher's answer key. A student who memorizes the key scores perfectly. A student who writes a brilliant original answer scores poorly because they used different words. BLEU works like that word-counting system. It is fine for translation where there are only a few right ways to say something, but terrible for open questions where creative, accurate answers are penalized for not matching the key.
Detailed answer & concept explanation~4 min readEverything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Everything important, quickly.
Open by identifying the metric mismatch: BLEU on open-ended questions. Explain what BLEU measures (n-gram precision) and why it works for translation but fails for open-ended generation. Walk through the three plausible explanations for the BLEU increase. Describe the metric stack that replaces BLEU for this task. Close with when BLEU remains appropriate.
Real products, models, and research that use this idea.
- BLEU remains the standard metric for machine translation evaluation at WMT (Workshop on Machine Translation), where constrained output spaces make n-gram overlap meaningful.
- AlpacaEval and Chatbot Arena use LLM-as-judge and human preference, not BLEU, for evaluating open-ended model quality, precisely because BLEU does not capture what matters.
- BERTScore was developed specifically to address BLEU's inability to handle paraphrases by using contextual embeddings instead of n-gram matching.
- Google's translation team uses BLEU alongside COMET and human evaluation, not BLEU alone, even for the constrained translation setting.
What an interviewer would ask next. Try answering before peeking at the approach.
Don't say thisRed flags and common mistakes that signal junior thinking. Click to expand.
Red flags and common mistakes that signal junior thinking. Click to expand.
Treating a BLEU improvement as proof of quality improvement on open-ended tasks where n-gram overlap does not correlate with answer quality.
The night-before-the-interview bullets. Scan these on the way to the call.
Primary sources. Skim if you want the original framing.
Same topic, related formats. Practice these next.