Reference: 'The cat sat on the mat.' Generated: 'A feline rested on the rug.' Predict whether BLEU-4 will be close to 0 or close to 1, and explain why.

Attempt it

You have a reference sentence: 'The cat sat on the mat.' and a generated sentence: 'A feline rested on the rug.' Compute or predict the BLEU-4 score. Will it be close to 0 or close to 1? Explain the reasoning in terms of n-gram overlap.

Quick reference

TL;DR

BLEU-4 is near zero because the sentences share zero 4-grams despite being semantically equivalent. BLEU measures n-gram overlap, not meaning.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Imagine you and a friend both describe the same picture, but you use completely different words. You say 'the cat sat on the mat' and your friend says 'a feline rested on the rug.' A teacher who checks your work by counting matching words would say you barely agree. But anyone looking at the picture would say you both described it perfectly. BLEU is like that word counting teacher. It checks how many matching word sequences you share, and since you used different words, your score is nearly zero even though you said the same thing.

Key concepts

BLEU Score N-gram Precision

Quick scan · 2–4 min

Everything important, quickly.

Open with the prediction: BLEU-4 near zero. Walk through the n-gram breakdown at each level (unigrams through 4-grams). Explain the geometric mean property. Name the core limitation: n-gram overlap does not capture semantic similarity. Contrast with BERTScore and BLEURT. Discuss task fit: constrained paraphrase tasks (MT) versus open paraphrase tasks (summarization, dialogue). Close on the production implication: never use BLEU alone on paraphrase rich tasks.

Real-world examples

Real products, models, and research that use this idea.

Machine translation benchmarks like WMT use BLEU alongside human evaluation, and the correlation between BLEU and human judgment drops sharply for language pairs with high paraphrase diversity.
BERTScore was introduced specifically to address BLEU's paraphrase blindness by computing cosine similarity between contextual token embeddings of the generated and reference texts.
Google's BLEURT is a learned metric that combines BLEU style features with BERT embeddings, trained to correlate with human quality judgments on paraphrase rich tasks.
Modern eval pipelines at production NLP companies typically use BLEU as one metric among several, pairing it with BERTScore or a judge model to capture both surface overlap and semantic quality.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QYou add 10 reference translations for the same sentence. Does BLEU-4 improve for the paraphrase, and if so, by how much?

Sign in or go Pro to see the approach for this follow-up.

QBERTScore would give a high score for this pair. When would BERTScore give a misleadingly high score?

Sign in or go Pro to see the approach for this follow-up.

QHow would you design a metric that captures both surface overlap and semantic similarity?

Sign in or go Pro to see the approach for this follow-up.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.