Bleu

A colleague presents this claim: 'Our chatbot improved because BLEU score went from 0.32 to 0.41 on our eval set of open-ended customer questions.' Spot the flaw in this reasoning.

BLEU score comes up in a meeting. What does it measure and when does it still make sense?

For a machine translation eval, a senior engineer says BLEU is fine as the primary metric. When are they right, and when should you push back?

Reference: 'The cat sat on the mat.' Generated: 'A feline rested on the rug.' Predict whether BLEU-4 will be close to 0 or close to 1, and explain why.