How does BERTScore improve on BLEU, and what is its key limitation?

MCQ·Medium·4.0 · 0·~1 min·Asked atNeo4j Pinterest Uipath·Relevant atHugging Face Microsoft

Attempt it

**BLEU** scores a generated sentence by counting how many short word sequences (n-grams) it shares with a reference. It is fast and language-agnostic, but it is brittle: it only rewards exact word matches. Say "physician" when the reference says "doctor" and BLEU gives you nothing, even though the meaning is identical. **BERTScore** fixes this by working with meaning instead of surface words. It runs both the candidate and the reference through a pretrained encoder like BERT, turning every token into a contextual embedding (a vector that captures meaning in context). It then matches each candidate token to its most similar reference token using cosine similarity, and aggregates those similarities into precision, recall, and an F1 score. Because embeddings place synonyms and paraphrases near each other, BERTScore rewards a good reworded answer that BLEU would penalize. That is the improvement: it captures semantic equivalence. The key limitation is that semantic similarity is not the same as truth. A response that confidently states a wrong fact in fluent, on-topic language sits close to a correct response in embedding space. BERTScore therefore reports a high score for a well-written hallucination, so it cannot be your only check for factual tasks.

**BERTScore** improves on BLEU by replacing exact n-gram overlap with soft semantic matching. It embeds every token of both candidate and reference through a contextual encoder. Then it greedily matches each token to its most similar counterpart by cosine similarity. From those matches it computes a precision (candidate tokens covered by the reference), a recall (reference tokens covered by the candidate), and their F1. Because synonyms and paraphrases land close in embedding space, a correctly reworded answer scores high. BLEU would penalize that same answer for not reusing the exact words. The improvement is concrete. BERTScore correlates far better with human judgment on paraphrase-heavy tasks like translation and summarization. It is robust to word order and lexical choice. The key limitation is the flip side of its strength. Semantic similarity does not imply factual correctness. A fluent response that states a wrong fact still sits near a correct response in the embedding space, so it earns a high score. The metric conflates style and topicality with truth. The practical takeaway: use BERTScore for fluency and semantic overlap, never as a standalone correctness signal. Pair it with grounded checks for factual tasks.

TL;DR

BERTScore matches contextual token embeddings by cosine similarity for a soft precision, recall, and F1, so it rewards paraphrase that BLEU misses. Its blind spot: a fluent wrong answer still scores high.

Memory aid

Easy to grasp

BLEU is a grader who only gives credit when your essay reuses the exact words of the answer key. Say 'a car' when the key says 'an automobile' and you lose points, even though you are right. BERTScore is a smarter grader. It understands that 'car' and 'automobile' mean nearly the same thing, so it gives near-full credit for good paraphrases. It does this by turning each word into a list of numbers that captures meaning, then matching your words to the key's words by closeness. The catch: this grader only checks whether your essay sounds like the key, not whether your facts are correct. Write a smooth, confident sentence that says something false, and it can still earn a high mark because it reads like a right answer.

Key concepts

LLM evaluation Embeddings Tokenization

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

BERTScore is the canonical answer to a basic failure of n-gram metrics: BLEU and ROUGE only give credit for exact word overlap, so they punish good paraphrases and reward shallow copying. Modern generation is full of valid rewordings, so a metric that cannot see "physician" and "doctor" as equivalent is a poor proxy for human judgment. The metric appeared in 2019 and quickly became a standard complement to ROUGE on summarization and translation leaderboards.

This deep dive walks the mechanism end to end: how BLEU works and where it breaks, how contextual embeddings let BERTScore match meaning instead of strings, the exact precision, recall, and F1 computation, the refinements that make the numbers usable, where learned metrics like BLEURT go further, and the one limitation every interviewer is probing for. Semantic similarity is not factual correctness, and a fluent hallucination can score as high as a true answer. That single sentence is the heart of why the right MCQ option pairs the paraphrase strength with the factual blind spot.

Where BLEU breaks: exact n-gram overlap

BLEU scores a candidate by computing modified n-gram precision against one or more references, typically for n-grams of length 1 through 4, then multiplying by a brevity penalty to discourage short outputs. ROUGE is its recall-oriented cousin used for summarization. Both are cheap, deterministic, and language-agnostic, which is why they dominated for two decades.

The problem is that they operate on surface strings. "The doctor examined the patient" and "The physician checked the patient" share little n-gram overlap despite being near-identical in meaning, so BLEU scores the paraphrase low. The metric also rewards the wrong behavior: a candidate that copies reference n-grams while garbling meaning can still score well.

For open-ended generation, translation, and summarization, this lexical rigidity makes BLEU correlate weakly with human ratings. The field needed a metric that grades meaning, not spelling.

There were earlier patches. Synonym-aware metrics like METEOR added stemming and WordNet synonym lookups, and ROUGE variants tweaked the matching unit. But these still leaned on hand-built resources and shallow lexical rules. They could not capture the deeper, context-dependent equivalence that embeddings make trivial, which is why the embedding-based approach took over once strong pretrained encoders were available.

The fix: contextual embeddings

Soft token-level F1: precision, recall, and the formula

Refinements: IDF weighting and baseline rescaling

Beyond cosine: learned metrics like BLEURT

The limitation interviewers probe: meaning is not truth

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Property	BLEU / ROUGE	BERTScore
Matching unit	Exact n-gram overlap	Contextual token embeddings
Paraphrase / synonyms	Penalized (no exact match)	Rewarded (close in vector space)
Word-order sensitivity	High (n-gram based)	Low (token-level matching)
Reference required	Yes	Yes
Catches factual error	No	No
Main cost	Brittle to wording	Encoder forward pass per token

Real products, models, and research that use this idea.

Hugging Face evaluate ships BERTScore as a standard text-generation metric, widely used for summarization leaderboards.
DeepEval and RAGAS expose embedding-similarity metrics but steer factual evals toward claim-level entailment instead.

What an interviewer would ask next. Try answering before peeking at the approach.

QHow exactly does BERTScore turn token-level cosine similarities into precision, recall, and F1?

Describe greedy matching: each candidate token takes its max similarity to any reference token (precision), each reference token takes its max to any candidate token (recall), then harmonic-mean them. Mention optional IDF weighting on the sums.

3 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Treating a high BERTScore as proof of correctness. It measures semantic overlap with a reference, not factual accuracy, so a fluent hallucination can score as high as a correct answer.

60 second bullets to scan on the way to the call.

Why exact n-gram overlap makes BLEU brittle to paraphrase
How contextual embeddings represent token meaning in context

Primary sources. Browse if you want the original framing.

Zhang et al., BERTScore: Evaluating Text Generation with BERT