How does BERTScore improve on BLEU, and what is its key limitation?

MCQ·Medium·4.0 · 0·~1 min·Relevant atHugging Face Microsoft

Attempt it

**BLEU** scores a generated sentence by counting how many short word sequences (n-grams) it shares with a reference. It is fast and language-agnostic, but it is brittle: it only rewards exact word matches. Say "physician" when the reference says "doctor" and BLEU gives you nothing, even though the meaning is identical. **BERTScore** fixes this by working with meaning instead of surface words. It runs both the candidate and the reference through a pretrained encoder like BERT, turning every token into a contextual embedding (a vector that captures meaning in context). It then matches each candidate token to its most similar reference token using cosine similarity, and aggregates those similarities into precision, recall, and an F1 score. Because embeddings place synonyms and paraphrases near each other, BERTScore rewards a good reworded answer that BLEU would penalize. That is the improvement: it captures semantic equivalence. The key limitation is that semantic similarity is not the same as truth. A response that confidently states a wrong fact in fluent, on-topic language sits close to a correct response in embedding space. BERTScore therefore reports a high score for a well-written hallucination, so it cannot be your only check for factual tasks.

**BERTScore** improves on BLEU by replacing exact n-gram overlap with soft semantic matching. It embeds every token of both candidate and reference through a contextual encoder. Then it greedily matches each token to its most similar counterpart by cosine similarity. From those matches it computes a precision (candidate tokens covered by the reference), a recall (reference tokens covered by the candidate), and their F1. Because synonyms and paraphrases land close in embedding space, a correctly reworded answer scores high. BLEU would penalize that same answer for not reusing the exact words. The improvement is concrete. BERTScore correlates far better with human judgment on paraphrase-heavy tasks like translation and summarization. It is robust to word order and lexical choice. The key limitation is the flip side of its strength. Semantic similarity does not imply factual correctness. A fluent response that states a wrong fact still sits near a correct response in the embedding space, so it earns a high score. The metric conflates style and topicality with truth. The practical takeaway: use BERTScore for fluency and semantic overlap, never as a standalone correctness signal. Pair it with grounded checks for factual tasks.

Quick reference

TL;DR

BERTScore matches contextual token embeddings by cosine similarity for a soft precision, recall, and F1, so it rewards paraphrase that BLEU misses. Its blind spot: a fluent wrong answer still scores high.

Memory aid

Easy to grasp

BLEU is a grader who only gives credit when your essay reuses the exact words of the answer key. Say 'a car' when the key says 'an automobile' and you lose points, even though you are right. BERTScore is a smarter grader. It understands that 'car' and 'automobile' mean nearly the same thing, so it gives near-full credit for good paraphrases. It does this by turning each word into a list of numbers that captures meaning, then matching your words to the key's words by closeness. The catch: this grader only checks whether your essay sounds like the key, not whether your facts are correct. Write a smooth, confident sentence that says something false, and it can still earn a high mark because it reads like a right answer.

Key concepts

LLM evaluation Embeddings Tokenization

Detailed answer & concept explanation

~7 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Quick scan · 2–4 min

Everything important, quickly.

5 min: BLEU brittleness, contextual embeddings, greedy cosine matching into precision/recall/F1, the factual-blindness limitation, and where BERTScore fits in a 2026 eval panel.

Side-by-side

Property	BLEU / ROUGE	BERTScore
Matching unit	Exact n-gram overlap	Contextual token embeddings
Paraphrase / synonyms	Penalized (no exact match)	Rewarded (close in vector space)
Word-order sensitivity	High (n-gram based)	Low (token-level matching)
Reference required	Yes	Yes
Catches factual error	No	No
Main cost	Brittle to wording	Encoder forward pass per token

Real-world examples

Real products, models, and research that use this idea.

Hugging Face evaluate ships BERTScore as a standard text-generation metric, widely used for summarization leaderboards.
DeepEval and RAGAS expose embedding-similarity metrics but steer factual evals toward claim-level entailment instead.
Translation and summarization benchmarks report BERTScore alongside ROUGE because it correlates better with human ratings.
Promptfoo includes similarity-based assertions so teams can gate outputs on semantic closeness to a reference answer.
Academic summarization papers in 2026 still report BERTScore F1 as a complement to human evaluation, not a replacement.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QHow exactly does BERTScore turn token-level cosine similarities into precision, recall, and F1?

QWhy does BERTScore need baseline rescaling, and what does the rescaled number represent?

QIf BERTScore cannot detect factual errors, how would you evaluate factual correctness instead?

QHow does encoder choice and layer selection change BERTScore results?

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Treating a high BERTScore as proof of correctness. It measures semantic overlap with a reference, not factual accuracy, so a fluent hallucination can score as high as a correct answer.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

References

Primary sources. Skim if you want the original framing.

Zhang et al., BERTScore: Evaluating Text Generation with BERT