You have a reference sentence: 'The cat sat on the mat.' and a generated sentence: 'A feline rested on the rug.' Compute or predict the BLEU-4 score. Will it be close to 0 or close to 1? Explain the reasoning in terms of n-gram overlap.
BLEU-4 is near zero because the sentences share zero 4-grams despite being semantically equivalent. BLEU measures n-gram overlap, not meaning.
Imagine you and a friend both describe the same picture, but you use completely different words. You say 'the cat sat on the mat' and your friend says 'a feline rested on the rug.' A teacher who checks your work by counting matching words would say you barely agree. But anyone looking at the picture would say you both described it perfectly. BLEU is like that word counting teacher. It checks how many matching word sequences you share, and since you used different words, your score is nearly zero even though you said the same thing.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
This example is a standard demonstration of BLEU's fundamental limitation: it measures surface level n-gram overlap, not semantic similarity. Two sentences that express identical meaning through different word choices score near zero on BLEU-4 because they share almost no multi-word sequences.
The exercise forces you to think about what BLEU actually computes and why that computation fails on paraphrases. Understanding this limitation is essential for choosing the right metric for your evaluation task and for interpreting BLEU scores correctly when you encounter them in papers and leaderboards.
The n-gram breakdown
Reference: 'The cat sat on the mat' (6 tokens). Generated: 'A feline rested on the rug' (6 tokens).
Unigrams: the generated tokens are {A, feline, rested, on, the, rug}. The reference tokens are {The, cat, sat, on, the, mat}. Matching (case insensitive): 'on' and 'the' appear in both. Unigram precision: 2/6 = 0.33.
Bigrams: the generated bigrams are {A feline, feline rested, rested on, on the, the rug}. The reference bigrams are {The cat, cat sat, sat on, on the, the mat}. Matching: 'on the'. Bigram precision: 1/5 = 0.20.
Trigrams: generated {A feline rested, feline rested on, rested on the, on the rug}. Reference {The cat sat, cat sat on, sat on the, on the mat}. Zero matches. Trigram precision: 0/4 = 0.
4-grams: generated {A feline rested on, feline rested on the, rested on the rug}. Reference {The cat sat on, cat sat on the, sat on the mat}. Zero matches. 4-gram precision: 0/3 = 0.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Machine translation benchmarks like WMT use BLEU alongside human evaluation, and the correlation between BLEU and human judgment drops sharply for language pairs with high paraphrase diversity.
- BERTScore was introduced specifically to address BLEU's paraphrase blindness by computing cosine similarity between contextual token embeddings of the generated and reference texts.
What an interviewer would ask next. Try answering before peeking at the approach.
QYou add 10 reference translations for the same sentence. Does BLEU-4 improve for the paraphrase, and if so, by how much?
Multi reference BLEU matches the generated text against each reference independently and takes the best match per n-gram. If one of the 10 references uses 'feline' and 'rug,' the overlap improves dramatically. This is why MT benchmarks use multi reference sets. However, 10 references still cannot cover the full paraphrase space, so BLEU remains an underestimate of quality for lexically diverse outputs.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Assuming BLEU captures semantic similarity, when it strictly measures n-gram overlap and assigns near zero to valid paraphrases.
60 second bullets to scan on the way to the call.
Why BLEU-4 is near zero for this pair despite semantic equivalence.
How to compute n-gram precision at each level (unigram through 4-gram).
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.