Explain the structural reason BLEU score is a poor metric for evaluating open-ended LLM outputs. Give a concrete example and suggest a better alternative.
Free · 2 AI evals / day
**BLEU** is a metric from machine translation. It measures how many short word sequences (n-grams) in the model's output also appear in a fixed reference answer, then multiplies in a penalty for outputs that are too short.
The core assumption is reference uniqueness: that there is essentially one correct way to phrase the answer. For translation that roughly holds, since good translations of a sentence look similar. For open-ended LLM tasks like chat, summarization, or explanation, it breaks badly.
Consider 'The cat is on the mat' versus 'A cat sits on the mat'. Same meaning, but they share only the bigram 'the mat', so BLEU is near zero. The second answer is correct, yet the metric calls it bad.
The structural problem: BLEU is a surface-form metric used on a meaning problem. It cannot give credit for synonyms, paraphrases, or reordering. With LLMs, where phrasing varies naturally, that penalty is systematic and has nothing to do with quality.
Better options grade meaning, not exact words. BERTScore compares contextual embeddings, BLEURT is trained against human ratings, and an LLM judge can score against a semantic rubric.
**BLEU** computes modified n-gram precision between an output and a fixed reference, then applies a brevity penalty. It rewards exact surface overlap.
The structural flaw is reference dependence. BLEU assumes there is one canonical phrasing. Open-ended generation violates this. There are many equally correct ways to answer, so a valid paraphrase shares few n-grams and scores near zero.
Concrete example. 'The cat is on the mat' and 'A cat sits on the mat' mean the same thing. They share only 'the mat' as a bigram, so BLEU collapses. A correct answer is scored as wrong.
The deeper issue: BLEU is a surface metric applied to a semantic equivalence problem. It penalizes synonyms, reordering, and style variation. For LLM output, that variation is the norm, so the penalty is systematic noise unrelated to quality. BLEU also has no notion of factuality or coherence.
Use meaning-aware metrics instead. BERTScore matches contextual embeddings and credits paraphrases. BLEURT is fine-tuned to correlate with human judgments. For truly open-ended work, an LLM judge with a semantic rubric is the 2026 default, with human spot-checks on a holdout.
Quick reference
TL;DR
BLEU scores surface n-gram overlap against a fixed reference. Open-ended outputs have many valid phrasings, so correct paraphrases score near zero. Use BERTScore or an LLM judge instead.
Imagine grading an essay by checking how many exact word-pairs match the teacher's model answer. A student who writes a perfect essay in their own words gets a low score, because their phrasing does not line up word for word with the key. That is what BLEU does: it counts shared chunks of words against one fixed answer. For a translation task that mostly works, since good translations look alike. But for open-ended writing there are a thousand correct ways to say the same thing. So a great answer that happens to use different words gets punished, and a clunky answer that parrots the reference gets rewarded. The fix is to grade meaning, not matching words: compare the ideas using embeddings (BERTScore) or have a smart model read both and judge.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Open-ended chat, summarization, or explanation where many phrasings are valid Correct paraphrases share few n-grams with the reference, so BLEU systematically penalizes good answers.
Any evaluation that must reflect factual accuracy or coherence BLEU only measures surface overlap and is blind to whether claims are true or logically consistent.
Comparing systems across different datasets or reference sets BLEU scores are not calibrated across tasks, so cross-dataset comparisons are meaningless and misleading.
Single-reference setups for tasks with high output diversity One reference cannot cover the space of valid answers, so overlap collapses for legitimate variation.
Quick scan · 2–4 min
Everything important, quickly.
5 min: what BLEU computes, the reference-uniqueness assumption, why open-ended generation breaks it, a concrete paraphrase example, and the meaning-aware alternatives plus when BLEU is still fine.
Real-world examples
Real products, models, and research that use this idea.
1Hugging Face evaluate still ships sacreBLEU for translation, but its summarization and chat tooling has moved to BERTScore and LLM-judge scores.
2RAGAS and TruLens skip BLEU entirely for RAG, using embedding similarity and claim-level entailment instead of n-gram overlap.
3Chatbot Arena ranks open-ended chat quality by human and LLM-judge pairwise Elo, not by any BLEU-style overlap metric.
4BLEURT and COMET are the learned metrics that WMT translation evaluation now favors over raw BLEU for human correlation.
5OpenAI Evals and Promptfoo default to model-graded (LLM-judge) checks for open-ended tasks rather than BLEU or ROUGE.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QBERTScore also compares against a reference. Why does it tolerate paraphrases when BLEU does not?
Sign in or go Pro to see the approach for this follow-up.
QWhen is BLEU still a reasonable metric to report?
Sign in or go Pro to see the approach for this follow-up.
QEmbedding and LLM-judge metrics are not free or deterministic. How do you make an open-ended eval reproducible?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Reporting BLEU as a quality score for chat or summarization outputs. It measures surface overlap with one reference, so correct paraphrases get penalized and the number tracks phrasing, not meaning.