BLEU score comes up in a meeting. What does it measure and when does it still make sense?

**BLEU** stands for Bilingual Evaluation Understudy. It was designed in the early 2000s to automatically evaluate machine translation, and it measures how many word sequences in the generated text also appear in a human reference translation. The mechanic is n-gram precision. BLEU looks at unigrams (single words), bigrams (pairs of consecutive words), trigrams, and 4-grams. For each n-gram size, it counts how many of the output's n-grams match the reference. The final score combines all four levels into a single number between 0 and 1. BLEU includes a brevity penalty. If the output is much shorter than the reference, the score is reduced. This prevents a model from gaming precision by producing only a few high-confidence words and stopping early. The one big task category where BLEU still works is machine translation. Translations have constrained phrasings, so a good translation naturally shares most of its n-grams with the reference. For open-ended generation like summarization, dialogue, or creative writing, BLEU is misleading. Many valid outputs use entirely different words from the reference, and BLEU penalizes them for it.

**BLEU** (Bilingual Evaluation Understudy) is a reference-based metric that measures n-gram precision: what fraction of the n-grams in the generated text also appear in the reference. It works at four levels simultaneously. Unigram precision checks individual word overlap. Bigram precision checks two-word sequence overlap. Trigram and 4-gram precision check longer phrase matches. The final BLEU score is a geometric mean of all four precision values, weighted equally by default. A brevity penalty reduces the score when the output is shorter than the reference, preventing short, safe outputs from getting artificially high precision. BLEU still makes sense for machine translation because the space of valid translations is tightly constrained. A correct English translation of a French sentence will naturally use many of the same words and phrases as a human reference. High n-gram overlap genuinely correlates with translation quality. For open-ended generation, BLEU breaks down. A summary that captures the main points in different words scores poorly. A dialogue response that is helpful but phrased differently from the reference gets punished. The metric cannot distinguish between a bad output and a good output that happens to use different vocabulary. In 2026, BLEU remains standard for translation evaluation but has been largely replaced by BERTScore, LLM-judge evaluation, and task-specific metrics for other generation tasks.