A colleague presents this claim: 'Our chatbot improved because BLEU score went from 0.32 to 0.41 on our eval set of open-ended customer questions.' Spot the flaw in this reasoning. · Zenaique

Attempt it

Click any words you think contain an error. Click again to unmark.

Mark at least one word to submit.

Quick reference

TL;DR

BLEU measures n-gram overlap and was designed for translation. On open-ended questions, excellent answers share few n-grams with references. A BLEU increase may reflect reference parroting, not genuine improvement.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Imagine grading an essay by counting how many exact word sequences match the teacher's answer key. A student who memorizes the key scores perfectly. A student who writes a brilliant original answer scores poorly because they used different words. BLEU works like that word-counting system. It is fine for translation where there are only a few right ways to say something, but terrible for open questions where creative, accurate answers are penalized for not matching the key.

Key concepts

BLEU Score Metric Mismatch Text Generation Metrics

Detailed answer & concept explanation

~4 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Sign in for 5 free unlocks/day or go Pro for unlimited. See Pro pricing.

Quick scan · 2–4 min

Everything important, quickly.

Open by identifying the metric mismatch: BLEU on open-ended questions. Explain what BLEU measures (n-gram precision) and why it works for translation but fails for open-ended generation. Walk through the three plausible explanations for the BLEU increase. Describe the metric stack that replaces BLEU for this task. Close with when BLEU remains appropriate.

Real-world examples

Real products, models, and research that use this idea.

BLEU remains the standard metric for machine translation evaluation at WMT (Workshop on Machine Translation), where constrained output spaces make n-gram overlap meaningful.
AlpacaEval and Chatbot Arena use LLM-as-judge and human preference, not BLEU, for evaluating open-ended model quality, precisely because BLEU does not capture what matters.
BERTScore was developed specifically to address BLEU's inability to handle paraphrases by using contextual embeddings instead of n-gram matching.
Google's translation team uses BLEU alongside COMET and human evaluation, not BLEU alone, even for the constrained translation setting.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QYou replace BLEU with BERTScore and the improvement vanishes. What does that tell you?

Sign in or go Pro to see the approach for this follow-up.

QCould you design a version of BLEU that works better for open-ended tasks?

Sign in or go Pro to see the approach for this follow-up.

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Treating a BLEU improvement as proof of quality improvement on open-ended tasks where n-gram overlap does not correlate with answer quality.

Sign in or go Pro to see all red flags and common mistakes. See Pro pricing.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.