Text Gen Metrics

A colleague presents this claim: 'Our chatbot improved because BLEU score went from 0.32 to 0.41 on our eval set of open-ended customer questions.' Spot the flaw in this reasoning.

ROUGE keeps showing up in summarization evals. How does it work and where does it break?

BLEU score comes up in a meeting. What does it measure and when does it still make sense?

When is ROUGE-L more appropriate than ROUGE-2 for evaluating LLM outputs?

When should you use token-level F1 vs exact match in a QA evaluation?

Explain the structural reason BLEU fails for open-ended LLM evaluation

Why does BLEU score poorly predict human judgment for open-ended LLM outputs?

Explain BERTScore's mechanism, its improvement over BLEU, and the specific failure mode that limits its use for factual LLM eval

How does BERTScore improve on BLEU, and what is its key limitation?