**BLEU** is an automatic metric built for machine translation. It scores an output by counting how many of its n-grams (1-word, 2-word, up to 4-word sequences) also appear in a reference answer. More overlap means a higher score.
That design has a hidden assumption: that the reference is the one canonical correct answer. For translation, that roughly holds, since good translations of a sentence look similar. For open-ended LLM tasks like summarization, question answering, or creative writing, it breaks badly.
The problem is paraphrase. A model can answer perfectly using synonyms or a different word order than the reference. BLEU sees almost no matching n-grams and assigns a near-zero score, even though a human would call the answer correct. That is exactly what option A describes.
The other options are wrong. BLEU has no minimum length rule. It measures precision, not recall, so option C reverses it. And it operates on tokenized text just fine, so tokenization is not the real issue. The real issue is that BLEU rewards surface wording, not meaning.
**BLEU** computes modified n-gram precision against one or more fixed references, then multiplies by a brevity penalty to discourage short outputs. It rewards surface overlap, not meaning.
The core failure for open-ended generation is the single-reference assumption. Tasks like summarization, dialogue, and QA have many valid phrasings. A correct answer that uses synonyms or reorders clauses shares few exact n-grams with the reference. BLEU then scores it near zero.
This is why option A is right. The metric measures lexical overlap, so a semantically correct paraphrase is punished. A fluent but wrong answer that happens to reuse reference words can score higher than a correct one.
The distractors are all false. BLEU has no minimum length requirement. It is a precision metric, not recall, so option C inverts it. Tokenization is orthogonal to its validity, so option D is a red herring.
The practical takeaway is to match metric to task. Use BLEU only where references are tight and exhaustive. For open-ended quality, prefer embedding-based metrics like BERTScore, or LLM-as-judge with human spot-checks.
Quick reference
TL;DR
BLEU rewards surface n-gram overlap with a fixed reference. A correct paraphrase using synonyms or different word order scores near zero, so BLEU tracks wording, not meaning.
Imagine grading an essay by checking how many exact phrases match the teacher's model answer, word for word. A student who writes a perfect essay in their own words gets a near-zero grade, because almost none of their phrases line up letter for letter with the model. Meanwhile a student who copies chunks but writes nonsense around them scores well. That is BLEU. It counts overlapping word sequences (n-grams) between the model's output and one reference answer. For translation, where good answers cluster tightly, that works okay. But for open-ended writing, summaries, or chatbot replies, there are countless correct ways to say the same thing. BLEU cannot see that two different sentences mean the same thing, so it punishes valid creativity and rewards copying.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Open-ended generation such as summarization, dialogue, or creative writing Many valid phrasings exist, so a correct paraphrase shares few n-grams and scores near zero despite being right.
Tasks where you only have a single reference answer per example BLEU assumes the reference is canonical. With one reference it cannot credit the many other correct wordings.
Sentence-level or per-example quality judgments BLEU was designed for corpus-level scoring and is noisy on single sentences, which misleads per-example decisions.
Evaluating semantic correctness or factual accuracy of an answer BLEU only sees lexical overlap, so it cannot detect whether the meaning or the facts are actually right.
Quick scan · 2–4 min
Everything important, quickly.
5 min: what BLEU computes (n-gram precision plus brevity penalty), the single-reference assumption, why paraphrase and synonyms break it on open-ended tasks, and which metrics to use instead.
Side-by-side
Metric
What it measures
Synonym credit
Best fit
BLEU
Clipped n-gram precision vs reference
No
Translation with tight references
ROUGE-L
Longest-common-subsequence recall
No
Summarization overlap, recall focus
BERTScore
Embedding cosine similarity of tokens
Yes
Open-ended semantic similarity
LLM-as-judge
Model-rated quality on a rubric
Yes
Nuanced open-ended quality
Real-world examples
Real products, models, and research that use this idea.
1WMT machine-translation evaluations still report BLEU and chrF as automatic baselines, but rank final systems with human direct-assessment scores.
2Hugging Face evaluate and sacreBLEU expose BLEU for translation, while pairing it with BERTScore for semantic similarity on generation tasks.
3RAGAS and similar 2026 eval stacks skip BLEU entirely for RAG answers, using claim-level faithfulness and answer relevance via LLM-as-judge.
4Summarization leaderboards moved from ROUGE and BLEU toward BERTScore, BLEURT, and LLM-judge scores because lexical overlap underrates good abstractive summaries.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QHow does BERTScore address the paraphrase problem that breaks BLEU?
Sign in or go Pro to see the approach for this follow-up.
QWhy does the geometric mean in BLEU make high-order n-gram failures so punishing?
Sign in or go Pro to see the approach for this follow-up.
QWhen is BLEU still the right tool to reach for in 2026?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Reporting BLEU as the headline quality metric for open-ended generation. BLEU measures lexical overlap with one reference, not semantic correctness, so paraphrases are punished and copying is rewarded.