ROUGE keeps showing up in summarization evals. How does it work and where does it break?

**ROUGE** stands for Recall-Oriented Understudy for Gisting Evaluation. It measures how much of the reference text's content appears in the generated output. This makes it the standard metric for summarization tasks. ROUGE comes in several variants. ROUGE-1 counts the overlap of single words (unigrams) between the output and reference. ROUGE-2 counts the overlap of two-word sequences (bigrams). ROUGE-L measures the longest common subsequence, which captures sentence-level structure beyond individual n-grams. The key difference from BLEU is the direction of comparison. BLEU measures precision: what fraction of the output's n-grams appear in the reference. ROUGE measures recall: what fraction of the reference's n-grams appear in the output. For summarization, recall is more important because you want the summary to capture the key information from the source, even if it includes extra explanatory text. A practical example: if the reference says 'the cat sat on the mat' and the output says 'the fluffy cat sat comfortably on the old mat near the door,' BLEU penalizes the extra words. ROUGE rewards the output because it contains all the reference's content.

**ROUGE** (Recall-Oriented Understudy for Gisting Evaluation) measures n-gram recall between a generated text and a reference, answering the question: how much of the reference content did the output capture? Three variants are standard. ROUGE-1 computes unigram recall. ROUGE-2 computes bigram recall, which captures phrase-level overlap. ROUGE-L computes the longest common subsequence (LCS) between the output and reference, rewarding outputs that preserve the order of the reference's content without requiring contiguous matches. The BLEU vs ROUGE distinction is precision vs recall. BLEU asks 'are the output's n-grams in the reference?' (precision). ROUGE asks 'are the reference's n-grams in the output?' (recall). For translation, precision matters because you want the output to be tight and accurate. For summarization, recall matters because you want the summary to include the key points from the source. ROUGE also reports an F1 variant that balances precision and recall. In practice, most summarization papers report ROUGE-1 F1, ROUGE-2 F1, and ROUGE-L F1, giving a multifaceted view of overlap. Like BLEU, ROUGE has the same fundamental limitation: it measures surface-level n-gram overlap, not meaning. A summary that paraphrases perfectly but uses different vocabulary scores low. For tasks requiring semantic evaluation, BERTScore or LLM-as-judge evaluation is a better choice.