Why does BLEU score poorly predict human judgment for open-ended LLM outputs?
BLEU rewards surface n-gram overlap with a fixed reference. A correct paraphrase using synonyms or different word order scores near zero, so BLEU tracks wording, not meaning.
Imagine grading an essay by checking how many exact phrases match the teacher's model answer, word for word. A student who writes a perfect essay in their own words gets a near-zero grade, because almost none of their phrases line up letter for letter with the model. Meanwhile a student who copies chunks but writes nonsense around them scores well. That is BLEU. It counts overlapping word sequences (n-grams) between the model's output and one reference answer. For translation, where good answers cluster tightly, that works okay. But for open-ended writing, summaries, or chatbot replies, there are countless correct ways to say the same thing. BLEU cannot see that two different sentences mean the same thing, so it punishes valid creativity and rewards copying.
Detailed answer & concept explanation~6 min readEverything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Everything important, quickly.
5 min: what BLEU computes (n-gram precision plus brevity penalty), the single-reference assumption, why paraphrase and synonyms break it on open-ended tasks, and which metrics to use instead.
| Metric | What it measures | Synonym credit | Best fit |
|---|---|---|---|
| BLEU | Clipped n-gram precision vs reference | No | Translation with tight references |
| ROUGE-L | Longest-common-subsequence recall | No | Summarization overlap, recall focus |
| BERTScore | Embedding cosine similarity of tokens | Yes | Open-ended semantic similarity |
| LLM-as-judge | Model-rated quality on a rubric | Yes | Nuanced open-ended quality |
Real products, models, and research that use this idea.
- WMT machine-translation evaluations still report BLEU and chrF as automatic baselines, but rank final systems with human direct-assessment scores.
- Hugging Face evaluate and sacreBLEU expose BLEU for translation, while pairing it with BERTScore for semantic similarity on generation tasks.
- RAGAS and similar 2026 eval stacks skip BLEU entirely for RAG answers, using claim-level faithfulness and answer relevance via LLM-as-judge.
- Summarization leaderboards moved from ROUGE and BLEU toward BERTScore, BLEURT, and LLM-judge scores because lexical overlap underrates good abstractive summaries.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow does BERTScore address the paraphrase problem that breaks BLEU?
QWhy does the geometric mean in BLEU make high-order n-gram failures so punishing?
Don't say thisRed flags and common mistakes that signal junior thinking. Click to expand.
Red flags and common mistakes that signal junior thinking. Click to expand.
Reporting BLEU as the headline quality metric for open-ended generation. BLEU measures lexical overlap with one reference, not semantic correctness, so paraphrases are punished and copying is rewarded.
The night-before-the-interview bullets. Scan these on the way to the call.
Primary sources. Skim if you want the original framing.
Same topic, related formats. Practice these next.