Explain the structural reason BLEU fails for open-ended LLM evaluation

**BLEU** is a metric from machine translation. It measures how many short word sequences (n-grams) in the model's output also appear in a fixed reference answer, then multiplies in a penalty for outputs that are too short. The core assumption is reference uniqueness: that there is essentially one correct way to phrase the answer. For translation that roughly holds, since good translations of a sentence look similar. For open-ended LLM tasks like chat, summarization, or explanation, it breaks badly. Consider 'The cat is on the mat' versus 'A cat sits on the mat'. Same meaning, but they share only the bigram 'the mat', so BLEU is near zero. The second answer is correct, yet the metric calls it bad. The structural problem: BLEU is a surface-form metric used on a meaning problem. It cannot give credit for synonyms, paraphrases, or reordering. With LLMs, where phrasing varies naturally, that penalty is systematic and has nothing to do with quality. Better options grade meaning, not exact words. BERTScore compares contextual embeddings, BLEURT is trained against human ratings, and an LLM judge can score against a semantic rubric.

**BLEU** computes modified n-gram precision between an output and a fixed reference, then applies a brevity penalty. It rewards exact surface overlap. The structural flaw is reference dependence. BLEU assumes there is one canonical phrasing. Open-ended generation violates this. There are many equally correct ways to answer, so a valid paraphrase shares few n-grams and scores near zero. Concrete example. 'The cat is on the mat' and 'A cat sits on the mat' mean the same thing. They share only 'the mat' as a bigram, so BLEU collapses. A correct answer is scored as wrong. The deeper issue: BLEU is a surface metric applied to a semantic equivalence problem. It penalizes synonyms, reordering, and style variation. For LLM output, that variation is the norm, so the penalty is systematic noise unrelated to quality. BLEU also has no notion of factuality or coherence. Use meaning-aware metrics instead. BERTScore matches contextual embeddings and credits paraphrases. BLEURT is fine-tuned to correlate with human judgments. For truly open-ended work, an LLM judge with a semantic rubric is the 2026 default, with human spot-checks on a holdout.