Reference: 'The cat sat on the mat.' Generated: 'A feline rested on the rug.' Predict whether BLEU-4 will be close to 0 or close to 1, and explain why.
You have a reference sentence: 'The cat sat on the mat.' and a generated sentence: 'A feline rested on the rug.' Compute or predict the BLEU-4 score. Will it be close to 0 or close to 1? Explain the reasoning in terms of n-gram overlap.
BLEU-4 is near zero because the sentences share zero 4-grams despite being semantically equivalent. BLEU measures n-gram overlap, not meaning.
Imagine you and a friend both describe the same picture, but you use completely different words. You say 'the cat sat on the mat' and your friend says 'a feline rested on the rug.' A teacher who checks your work by counting matching words would say you barely agree. But anyone looking at the picture would say you both described it perfectly. BLEU is like that word counting teacher. It checks how many matching word sequences you share, and since you used different words, your score is nearly zero even though you said the same thing.
Detailed answer & concept explanation~4 min readEverything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Everything important, quickly.
Open with the prediction: BLEU-4 near zero. Walk through the n-gram breakdown at each level (unigrams through 4-grams). Explain the geometric mean property. Name the core limitation: n-gram overlap does not capture semantic similarity. Contrast with BERTScore and BLEURT. Discuss task fit: constrained paraphrase tasks (MT) versus open paraphrase tasks (summarization, dialogue). Close on the production implication: never use BLEU alone on paraphrase rich tasks.
Real products, models, and research that use this idea.
- Machine translation benchmarks like WMT use BLEU alongside human evaluation, and the correlation between BLEU and human judgment drops sharply for language pairs with high paraphrase diversity.
- BERTScore was introduced specifically to address BLEU's paraphrase blindness by computing cosine similarity between contextual token embeddings of the generated and reference texts.
- Google's BLEURT is a learned metric that combines BLEU style features with BERT embeddings, trained to correlate with human quality judgments on paraphrase rich tasks.
- Modern eval pipelines at production NLP companies typically use BLEU as one metric among several, pairing it with BERTScore or a judge model to capture both surface overlap and semantic quality.
What an interviewer would ask next. Try answering before peeking at the approach.
QYou add 10 reference translations for the same sentence. Does BLEU-4 improve for the paraphrase, and if so, by how much?
QBERTScore would give a high score for this pair. When would BERTScore give a misleadingly high score?
Don't say thisRed flags and common mistakes that signal junior thinking. Click to expand.
Red flags and common mistakes that signal junior thinking. Click to expand.
Assuming BLEU captures semantic similarity, when it strictly measures n-gram overlap and assigns near zero to valid paraphrases.
The night-before-the-interview bullets. Scan these on the way to the call.
Primary sources. Skim if you want the original framing.
Same topic, related formats. Practice these next.