ROUGE-L, BERTScore, and LLM-as-judge are on the table for a summarization eval. Which covers what, and which drops first if budget is tight?
Drop ROUGE-L first. BERTScore subsumes lexical overlap with semantic similarity, and LLM-as-judge adds coherence and faithfulness that no automated metric captures.
Imagine grading a book report three ways: counting shared phrases with the original (ROUGE-L), checking if the meaning is the same even with different words (BERTScore), and having a teacher read it for quality (LLM-as-judge). If you can only afford two, drop the phrase-counting. The meaning-checker already catches what the phrase-counter catches, plus more. And you definitely need the teacher, because only they can tell if the report makes sense as a whole.
Detailed answer & concept explanation~3 min readEverything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Everything important, quickly.
Open by explaining what each metric measures at its mechanism level. Show that BERTScore subsumes ROUGE-L for paraphrase-heavy summarization. Explain why LLM-as-judge adds unique dimensions (coherence, faithfulness, completeness). Walk the budget tradeoff: drop ROUGE-L, keep BERTScore as fast automated signal and LLM-as-judge on a sample. Close with the production setup.
Real products, models, and research that use this idea.
- ROUGE remains the standard metric on the CNN/DailyMail benchmark for historical continuity, but production summarization systems increasingly report BERTScore or LLM-as-judge scores alongside ROUGE.
- AlpacaEval uses LLM-as-judge rather than ROUGE for summarization tasks because rubric-based evaluation captures quality dimensions that reference-similarity metrics miss.
- BERTScore was specifically designed to address ROUGE's inability to handle paraphrases, using contextual embeddings from BERT to match tokens by meaning rather than string identity.
- Braintrust and Promptfoo both support summarization evaluation with configurable metric stacks that prioritize LLM-as-judge and BERTScore over ROUGE.
What an interviewer would ask next. Try answering before peeking at the approach.
QBERTScore uses which BERT layer matters for the embedding comparison. How do you choose?
Don't say thisRed flags and common mistakes that signal junior thinking. Click to expand.
Red flags and common mistakes that signal junior thinking. Click to expand.
Keeping ROUGE-L and dropping LLM-as-judge, losing the only metric that measures coherence, faithfulness, and completeness.
The night-before-the-interview bullets. Scan these on the way to the call.
Primary sources. Skim if you want the original framing.
Same topic, related formats. Practice these next.