Explain the structural reason BLEU fails for open ended LLM evaluation
Explain the structural reason BLEU score is a poor metric for evaluating open ended LLM outputs. Give a concrete example and suggest a better alternative.
BLEU scores surface n-gram overlap against a fixed reference. Open-ended outputs have many valid phrasings, so correct paraphrases score near zero. Use BERTScore or an LLM judge instead.
Imagine grading an essay by checking how many exact word-pairs match the teacher's model answer. A student who writes a perfect essay in their own words gets a low score, because their phrasing does not line up word for word with the key. That is what BLEU does: it counts shared chunks of words against one fixed answer. For a translation task that mostly works, since good translations look alike. But for open-ended writing there are a thousand correct ways to say the same thing. So a great answer that happens to use different words gets punished, and a clunky answer that parrots the reference gets rewarded. The fix is to grade meaning, not matching words: compare the ideas using embeddings (BERTScore) or have a smart model read both and judge.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
BLEU (Bilingual Evaluation Understudy) is one of the oldest automatic text-generation metrics, introduced for machine translation in 2002. It is fast, cheap, deterministic, and language-agnostic, which is why it spread far beyond the task it was designed for. By 2026 it is still the reflexive default many engineers reach for when they need a number for any text-generation system.
That reflex is the problem. BLEU answers a narrow question: how much does this output's surface wording overlap with a fixed reference, and that question is only a good proxy for quality when the space of correct outputs is small and lexically similar. Open-ended LLM tasks violate exactly that condition. This deep dive walks what BLEU actually computes, the structural assumption it bakes in, why open-ended generation breaks it, and what to use instead.
What BLEU actually computes
BLEU is built from modified n-gram precision. For each n from 1 to 4 it counts how many n-grams in the candidate output also appear in the reference, clips each count by the reference's own count (so repeating a word cannot inflate the score), and divides by the total n-grams in the candidate. It then takes the geometric mean across the four orders and multiplies by a brevity penalty that punishes outputs shorter than the reference.
The full corpus-level score is the canonical formula every translation paper cites:
Here p_n is the clipped n-gram precision for order n, w_n is its weight (usually uniform at 1/4), and BP is the brevity penalty. The geometric mean means any single order scoring zero drives the whole product to zero. That is why a short paraphrase with no 4-gram overlap can collapse to near zero even when its unigrams match.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Hugging Face evaluate still ships sacreBLEU for translation, but its summarization and chat tooling has moved to BERTScore and LLM-judge scores.
- RAGAS and TruLens skip BLEU entirely for RAG, using embedding similarity and claim-level entailment instead of n-gram overlap.
What an interviewer would ask next. Try answering before peeking at the approach.
QBERTScore also compares against a reference. Why does it tolerate paraphrases when BLEU does not?
Contrast surface n-gram matching with contextual-embedding cosine matching. BERTScore aligns each output token to its closest reference token in embedding space, so synonyms and reordering still match. It aligns greedily rather than requiring exact contiguous overlap.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Reporting BLEU as a quality score for chat or summarization outputs. It measures surface overlap with one reference, so correct paraphrases get penalized and the number tracks phrasing, not meaning.
60 second bullets to scan on the way to the call.
What BLEU actually computes: modified n-gram precision plus brevity penalty
The reference uniqueness assumption and why it holds for translation
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.