BLEU score comes up in a meeting. What does it measure and when does it still make sense?

Flashcard·Easy·4.0 · 0·~30s

Attempt it

BLEU (Bilingual Evaluation Understudy) measures n-gram precision between a generated text and one or more reference translations. It counts what fraction of the n-grams in the output also appear in the reference, with a brevity penalty to discourage short outputs that game precision. BLEU still makes sense for machine translation because translations have constrained valid phrasings. For open-ended generation, BLEU is misleading because many valid outputs share few n-grams with the reference.

Your answer

**BLEU** stands for Bilingual Evaluation Understudy. It was designed in the early 2000s to automatically evaluate machine translation, and it measures how many word sequences in the generated text also appear in a human reference translation. The mechanic is n-gram precision. BLEU looks at unigrams (single words), bigrams (pairs of consecutive words), trigrams, and 4-grams. For each n-gram size, it counts how many of the output's n-grams match the reference. The final score combines all four levels into a single number between 0 and 1. BLEU includes a brevity penalty. If the output is much shorter than the reference, the score is reduced. This prevents a model from gaming precision by producing only a few high-confidence words and stopping early. The one big task category where BLEU still works is machine translation. Translations have constrained phrasings, so a good translation naturally shares most of its n-grams with the reference. For open-ended generation like summarization, dialogue, or creative writing, BLEU is misleading. Many valid outputs use entirely different words from the reference, and BLEU penalizes them for it.

**BLEU** (Bilingual Evaluation Understudy) is a reference-based metric that measures n-gram precision: what fraction of the n-grams in the generated text also appear in the reference. It works at four levels simultaneously. Unigram precision checks individual word overlap. Bigram precision checks two-word sequence overlap. Trigram and 4-gram precision check longer phrase matches. The final BLEU score is a geometric mean of all four precision values, weighted equally by default. A brevity penalty reduces the score when the output is shorter than the reference, preventing short, safe outputs from getting artificially high precision. BLEU still makes sense for machine translation because the space of valid translations is tightly constrained. A correct English translation of a French sentence will naturally use many of the same words and phrases as a human reference. High n-gram overlap genuinely correlates with translation quality. For open-ended generation, BLEU breaks down. A summary that captures the main points in different words scores poorly. A dialogue response that is helpful but phrased differently from the reference gets punished. The metric cannot distinguish between a bad output and a good output that happens to use different vocabulary. In 2026, BLEU remains standard for translation evaluation but has been largely replaced by BERTScore, LLM-judge evaluation, and task-specific metrics for other generation tasks.

Quick reference

TL;DR

BLEU measures n-gram precision between generated and reference text, designed for machine translation where valid outputs share phrasing. Misleading for open-ended generation.

Memory aid

Easy to grasp

Imagine you asked ten translators to translate the same paragraph from French to English. Their translations would use mostly the same words in mostly the same order, because there are only so many natural ways to say the same thing in English. BLEU takes advantage of this. It looks at the machine's translation and checks how many of its word sequences (single words, pairs, triples, groups of four) also appear in the human reference translation. The more matches, the higher the score. There is also a penalty for outputs that are too short, since a very short translation could get all its words right but miss half the meaning. This works well for translation because the outputs are constrained. But if you asked ten people to write a creative story about a cat, they would use completely different words. BLEU would score all of them poorly against any single reference, even though they are all good stories.

Key concepts

BLEU N-gram Precision Machine Translation Brevity Penalty Reference-Based

Detailed answer & concept explanation

~5 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Quick scan · 2–4 min

Everything important, quickly.

5 min: define BLEU as n-gram precision with brevity penalty, explain the four n-gram levels, name translation as the valid use case, explain why it fails on open-ended generation, and name BERTScore and LLM-judge as replacements.

Real-world examples

Real products, models, and research that use this idea.

Google Translate reports BLEU scores internally on its translation benchmarks, and BLEU improvements correlate with user preference gains for the translation use case.
The WMT (Workshop on Machine Translation) annual shared task uses BLEU as one of its primary evaluation metrics alongside human evaluation.
Hugging Face's evaluate library includes a BLEU implementation that is commonly used for translation model comparisons in open-source research.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QHow does modified n-gram precision differ from standard n-gram precision, and why does the modification matter?

QWhen would you use BLEU with multiple references instead of a single reference?

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Applying BLEU to open-ended generation tasks like summarization or dialogue. BLEU penalizes valid paraphrases because it only counts exact n-gram matches, making it a poor proxy for quality on tasks with diverse valid outputs.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

References

Primary sources. Skim if you want the original framing.

Papineni et al., BLEU: a Method for Automatic Evaluation of Machine Translation