Explain BERTScore's mechanism, its improvement over BLEU, and the specific failure mode that limits its use for factual LLM eval

**BERTScore** is a metric for comparing a generated text against a reference text. Instead of checking whether the exact same words appear, it checks whether the words mean the same thing. It works by turning every token into a contextual embedding using a model like BERT. Then, for each token in the candidate output, it finds the most similar token in the reference using cosine similarity. It averages those best-match scores to get a recall number, does the reverse for precision, and combines them into an F1 score. The big improvement over BLEU is robustness to paraphrase. BLEU counts overlapping word sequences, so 'The fast dog' and 'The rapid dog' look completely different to it. BERTScore sees that 'fast' and 'rapid' have nearly identical embeddings, so it scores them as a good match. The catch is the failure mode. Similar-looking words do not mean the facts are right. A wrong answer that uses the correct topic vocabulary can still score high. So BERTScore tells you whether two texts are about the same thing in the same style, not whether the claims inside are true.

**BERTScore** scores a candidate against a reference using contextual embeddings rather than exact word overlap. Every token in both texts is embedded with a model like BERT. Each candidate token is then greedily matched to its most similar reference token by cosine similarity. Recall averages the best-match score over reference tokens; precision averages it over candidate tokens; F1 combines the two. An optional IDF weighting down-weights common function words so content words dominate. The improvement over BLEU is paraphrase robustness. BLEU counts n-gram overlap, so 'The rapid dog' and 'The fast dog' share no bigrams and score poorly. BERTScore sees that 'rapid' and 'fast' sit close in embedding space and scores them as a strong match. It also tolerates valid reorderings and synonyms that wreck BLEU. The limitation that matters for factual eval is this. Contextual embeddings capture semantic and stylistic similarity, not truth. A hallucinated 'Battle of Hastings was in 1166' scores high against a 1066 reference, since the topic and vocabulary match. BERTScore conflates sounding right with being right, so it cannot police factual grounding.