ROUGE-L, BERTScore, and LLM-as-judge are on the table for a summarization eval. Which covers what, and which drops first if budget is tight? · Zenaique

Attempt it

Quick reference

TL;DR

Drop ROUGE-L first. BERTScore subsumes lexical overlap with semantic similarity, and LLM-as-judge adds coherence and faithfulness that no automated metric captures.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Imagine grading a book report three ways: counting shared phrases with the original (ROUGE-L), checking if the meaning is the same even with different words (BERTScore), and having a teacher read it for quality (LLM-as-judge). If you can only afford two, drop the phrase-counting. The meaning-checker already catches what the phrase-counter catches, plus more. And you definitely need the teacher, because only they can tell if the report makes sense as a whole.

Key concepts

ROUGE BERTScore LLM-as-Judge

Detailed answer & concept explanation

~3 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Sign in for 5 free unlocks/day or go Pro for unlimited. See Pro pricing.

Quick scan · 2–4 min

Everything important, quickly.

Open by explaining what each metric measures at its mechanism level. Show that BERTScore subsumes ROUGE-L for paraphrase-heavy summarization. Explain why LLM-as-judge adds unique dimensions (coherence, faithfulness, completeness). Walk the budget tradeoff: drop ROUGE-L, keep BERTScore as fast automated signal and LLM-as-judge on a sample. Close with the production setup.

Real-world examples

Real products, models, and research that use this idea.

ROUGE remains the standard metric on the CNN/DailyMail benchmark for historical continuity, but production summarization systems increasingly report BERTScore or LLM-as-judge scores alongside ROUGE.
AlpacaEval uses LLM-as-judge rather than ROUGE for summarization tasks because rubric-based evaluation captures quality dimensions that reference-similarity metrics miss.
BERTScore was specifically designed to address ROUGE's inability to handle paraphrases, using contextual embeddings from BERT to match tokens by meaning rather than string identity.
Braintrust and Promptfoo both support summarization evaluation with configurable metric stacks that prioritize LLM-as-judge and BERTScore over ROUGE.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QBERTScore uses which BERT layer matters for the embedding comparison. How do you choose?

Sign in or go Pro to see the approach for this follow-up.

QYour LLM-as-judge scores faithfulness at 95% but users report hallucinated details in production summaries. What went wrong?

Sign in or go Pro to see the approach for this follow-up.

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Keeping ROUGE-L and dropping LLM-as-judge, losing the only metric that measures coherence, faithfulness, and completeness.

Sign in or go Pro to see all red flags and common mistakes. See Pro pricing.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.