For a machine translation eval, a senior engineer says BLEU is fine as the primary metric. When are they right, and when should you push back?
BLEU works for constrained translation (technical, legal) with high-quality references. Push back for creative content where valid translations diverge, and add COMET or BERTScore for semantic equivalence.
Imagine translating an instruction manual versus translating a poem. The manual has one right way to say each step, so checking word overlap with the answer key works. The poem has a hundred ways to express the same feeling, so checking word overlap penalizes beautiful translations that use different words. BLEU is the word-overlap checker. It works for the manual but fails for the poem.
Detailed answer & concept explanation~3 min readEverything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Everything important, quickly.
Open with the output-entropy framing: constrained tasks have tight valid-output distributions where BLEU works; creative tasks have broad distributions where it fails. Walk when the senior engineer is right (technical, legal) and when to push back (marketing, dialog). Introduce COMET and BERTScore as complements. Close with the production stack: BLEU for fast CI, COMET for quality, human eval for calibration.
Real products, models, and research that use this idea.
- WMT (Workshop on Machine Translation) reports BLEU alongside COMET and human evaluation scores, recognizing that BLEU alone is insufficient for creative content.
- Google Translate's internal evaluation uses BLEU for regression detection across all language pairs and COMET for quality assessment on creative and marketing content.
- COMET was developed by Unbabel and trained on millions of human quality annotations, making it the strongest learned metric for translation evaluation as of 2026.
- DeepL uses a combination of automated metrics and human post-editing rates, with BLEU serving as a fast CI check rather than the primary quality signal.
What an interviewer would ask next. Try answering before peeking at the approach.
QCOMET is trained on human judgments. What happens when the human judgment distribution shifts (different annotator pool, different quality standards)?
Don't say thisRed flags and common mistakes that signal junior thinking. Click to expand.
Red flags and common mistakes that signal junior thinking. Click to expand.
Applying BLEU uniformly across all translation tasks without considering whether the output space is constrained (technical) or open (creative).
The night-before-the-interview bullets. Scan these on the way to the call.
Primary sources. Skim if you want the original framing.
Same topic, related formats. Practice these next.