Why does BLEU score poorly predict human judgment for open ended LLM outputs?
BLEU rewards surface n-gram overlap with a fixed reference. A correct paraphrase using synonyms or different word order scores near zero, so BLEU tracks wording, not meaning.
Imagine grading an essay by checking how many exact phrases match the teacher's model answer, word for word. A student who writes a perfect essay in their own words gets a near-zero grade, because almost none of their phrases line up letter for letter with the model. Meanwhile a student who copies chunks but writes nonsense around them scores well. That is BLEU. It counts overlapping word sequences (n-grams) between the model's output and one reference answer. For translation, where good answers cluster tightly, that works okay. But for open-ended writing, summaries, or chatbot replies, there are countless correct ways to say the same thing. BLEU cannot see that two different sentences mean the same thing, so it punishes valid creativity and rewards copying.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
BLEU (Bilingual Evaluation Understudy) is the original automatic metric for text generation, introduced in 2002 for machine translation. Its appeal is obvious: it is cheap, fast, deterministic, and needs no model at inference time, only a reference string and some n-gram counting. For two decades it was the default number on translation leaderboards.
The trouble is that people kept reaching for BLEU on tasks it was never built for. Summarization, question answering, dialogue, and creative generation are all open-ended, with many equally correct ways to phrase an answer. This deep dive walks the exact mechanics of BLEU, shows precisely where its single-reference assumption breaks, explains why the distractor options in the question are wrong, and lays out what to use instead in a modern eval stack.
What BLEU actually computes
BLEU scores a candidate output by measuring n-gram precision against one or more reference strings. For each n from 1 to 4, it counts how many candidate n-grams appear in a reference, clips each count at the maximum times that n-gram appears in any reference, and divides by the total candidate n-grams. Clipping stops a model from gaming the score by repeating one high-frequency word.
Those per-order precisions are then combined with a geometric mean, and the result is multiplied by a brevity penalty that punishes outputs shorter than the reference. The full formula is:
Here p_n is the clipped precision for n-grams of order n, w_n are weights that usually sum to 1, and BP is the brevity penalty. The brevity penalty is BP = 1 when the candidate is at least as long as the reference, and BP = exp(1 - r/c) when the candidate length c is shorter than the reference length r. The geometric mean of the precisions is exactly what the exponential of the weighted log sum expresses.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
| Metric | What it measures | Synonym credit | Best fit |
|---|---|---|---|
| BLEU | Clipped n-gram precision vs reference | No | Translation with tight references |
| ROUGE-L | Longest-common-subsequence recall | No | Summarization overlap, recall focus |
| BERTScore | Embedding cosine similarity of tokens | Yes | Open-ended semantic similarity |
| LLM-as-judge | Model-rated quality on a rubric | Yes | Nuanced open-ended quality |
Real products, models, and research that use this idea.
- WMT machine-translation evaluations still report BLEU and chrF as automatic baselines, but rank final systems with human direct-assessment scores.
- Hugging Face evaluate and sacreBLEU expose BLEU for translation, while pairing it with BERTScore for semantic similarity on generation tasks.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow does BERTScore address the paraphrase problem that breaks BLEU?
Explain that BERTScore embeds tokens and matches them by cosine similarity, so synonyms and reworded phrases align in vector space. Note the cost: it needs an embedding model and can still miss factual errors that read fluently.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Reporting BLEU as the headline quality metric for open-ended generation. BLEU measures lexical overlap with one reference, not semantic correctness, so paraphrases are punished and copying is rewarded.
60 second bullets to scan on the way to the call.
What BLEU actually measures and the n-gram precision definition
The brevity penalty and why it exists
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.