How does BERTScore improve on BLEU, and what is its key limitation?

Attempt it

Quick reference

TL;DR

BERTScore matches contextual token embeddings by cosine similarity for a soft precision, recall, and F1, so it rewards paraphrase that BLEU misses. Its blind spot: a fluent wrong answer still scores high.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

BLEU is a grader who only gives credit when your essay reuses the exact words of the answer key. Say 'a car' when the key says 'an automobile' and you lose points, even though you are right. BERTScore is a smarter grader. It understands that 'car' and 'automobile' mean nearly the same thing, so it gives near-full credit for good paraphrases. It does this by turning each word into a list of numbers that captures meaning, then matching your words to the key's words by closeness. The catch: this grader only checks whether your essay sounds like the key, not whether your facts are correct. Write a smooth, confident sentence that says something false, and it can still earn a high mark because it reads like a right answer.

Key concepts

LLM evaluation Embeddings Tokenization

Quick scan · 2–4 min

Everything important, quickly.

5 min: BLEU brittleness, contextual embeddings, greedy cosine matching into precision/recall/F1, the factual-blindness limitation, and where BERTScore fits in a 2026 eval panel.

Side-by-side

Property	BLEU / ROUGE	BERTScore
Matching unit	Exact n-gram overlap	Contextual token embeddings
Paraphrase / synonyms	Penalized (no exact match)	Rewarded (close in vector space)
Word-order sensitivity	High (n-gram based)	Low (token-level matching)
Reference required	Yes	Yes
Catches factual error	No	No
Main cost	Brittle to wording	Encoder forward pass per token

Real-world examples

Real products, models, and research that use this idea.

Hugging Face evaluate ships BERTScore as a standard text-generation metric, widely used for summarization leaderboards.
DeepEval and RAGAS expose embedding-similarity metrics but steer factual evals toward claim-level entailment instead.
Translation and summarization benchmarks report BERTScore alongside ROUGE because it correlates better with human ratings.
Promptfoo includes similarity-based assertions so teams can gate outputs on semantic closeness to a reference answer.
Academic summarization papers in 2026 still report BERTScore F1 as a complement to human evaluation, not a replacement.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QHow exactly does BERTScore turn token-level cosine similarities into precision, recall, and F1?

Sign in or go Pro to see the approach for this follow-up.

QWhy does BERTScore need baseline rescaling, and what does the rescaled number represent?

Sign in or go Pro to see the approach for this follow-up.

QIf BERTScore cannot detect factual errors, how would you evaluate factual correctness instead?

Sign in or go Pro to see the approach for this follow-up.

QHow does encoder choice and layer selection change BERTScore results?

Sign in or go Pro to see the approach for this follow-up.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.

Zhang et al., BERTScore: Evaluating Text Generation with BERT