Drop ROUGE-L first. BERTScore subsumes lexical overlap with semantic similarity, and LLM-as-judge adds coherence and faithfulness that no automated metric captures.
Imagine grading a book report three ways: counting shared phrases with the original (ROUGE-L), checking if the meaning is the same even with different words (BERTScore), and having a teacher read it for quality (LLM-as-judge). If you can only afford two, drop the phrase-counting. The meaning-checker already catches what the phrase-counter catches, plus more. And you definitely need the teacher, because only they can tell if the report makes sense as a whole.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Metric selection for summarization evaluation is a layering problem. Each metric captures a different dimension of summary quality, and the budget question forces you to identify which layers add the most unique signal.
The three candidates measure fundamentally different things. ROUGE-L measures lexical recall via longest common subsequence. BERTScore measures semantic similarity via contextual embeddings. LLM-as-judge measures coherence, faithfulness, and completeness via rubric. The question is which pair preserves the most diagnostic value.
ROUGE-L: lexical recall via longest common subsequence
ROUGE-L computes the longest common subsequence (LCS) between the generated summary and the reference summary. The LCS captures word-level overlap that respects order but allows gaps. A summary that uses the same words in roughly the same order as the reference scores high.
The limitation is fundamental: ROUGE-L penalizes paraphrasing. Modern abstractive summarization models routinely rephrase source content, which is desirable behavior. A summary that says "the company's revenue increased by 15%" scores zero against a reference that says "sales grew fifteen percent" despite conveying identical information.
ROUGE-L remains the standard metric on benchmarks like CNN/DailyMail and XSum because it was there first and published results use it. For production evaluation, its signal is mostly subsumed by BERTScore.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- ROUGE remains the standard metric on the CNN/DailyMail benchmark for historical continuity, but production summarization systems increasingly report BERTScore or LLM-as-judge scores alongside ROUGE.
- AlpacaEval uses LLM-as-judge rather than ROUGE for summarization tasks because rubric-based evaluation captures quality dimensions that reference-similarity metrics miss.
What an interviewer would ask next. Try answering before peeking at the approach.
QBERTScore uses which BERT layer matters for the embedding comparison. How do you choose?
Different layers capture different levels of abstraction. Lower layers capture syntax; higher layers capture semantics. For summarization, use a higher layer (layer 17 to 24 for BERT-large) because you care about meaning, not surface form. The BERTScore library has recommended layer settings per model.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Keeping ROUGE-L and dropping LLM-as-judge, losing the only metric that measures coherence, faithfulness, and completeness.
60 second bullets to scan on the way to the call.
What ROUGE-L measures at a mechanism level (longest common subsequence)
What BERTScore measures (semantic similarity via contextual embeddings)
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.