For a machine translation eval, a senior engineer says BLEU is fine as the primary metric. When are they right, and when should you push back?
BLEU works for constrained translation (technical, legal) with high-quality references. Push back for creative content where valid translations diverge, and add COMET or BERTScore for semantic equivalence.
Imagine translating an instruction manual versus translating a poem. The manual has one right way to say each step, so checking word overlap with the answer key works. The poem has a hundred ways to express the same feeling, so checking word overlap penalizes beautiful translations that use different words. BLEU is the word-overlap checker. It works for the manual but fails for the poem.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
The question tests whether you can apply a metric contextually rather than dogmatically. BLEU is neither always right nor always wrong for translation. Its validity depends on the relationship between the content type and the metric's assumptions.
BLEU assumes that n-gram overlap with a reference is a good proxy for translation quality. This assumption holds when the valid output space is narrow (technical content) and fails when the valid output space is broad (creative content). The rest of this explanation develops both cases and introduces the complementary metrics that fill BLEU's gaps.
When BLEU works: constrained translation
Technical documentation, legal contracts, medical records, software interfaces. These domains have fixed terminology, conventional sentence structures, and a narrow set of valid translations for each source sentence. "Blood pressure" translates to a single term in most target languages. "Section 4.2 of the agreement" has one correct rendering.
In these settings, a correct translation shares most 4-grams with a professional reference. BLEU's n-gram precision measure correlates well with human quality judgments. The metric is fast (milliseconds per sentence), well-understood (decades of calibration at WMT), and requires no GPU or model inference.
The senior engineer is right to call BLEU sufficient here, with two caveats: the references must be from professional translators in the same domain, and the language pair must be well-represented in the reference set.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- WMT (Workshop on Machine Translation) reports BLEU alongside COMET and human evaluation scores, recognizing that BLEU alone is insufficient for creative content.
- Google Translate's internal evaluation uses BLEU for regression detection across all language pairs and COMET for quality assessment on creative and marketing content.
What an interviewer would ask next. Try answering before peeking at the approach.
QCOMET is trained on human judgments. What happens when the human judgment distribution shifts (different annotator pool, different quality standards)?
COMET's calibration drifts. Re-train or fine-tune on your annotator pool's judgments. If that is impractical, use COMET as a relative ranking tool (which translation is better) rather than an absolute score, because relative rankings are more robust to calibration drift.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Applying BLEU uniformly across all translation tasks without considering whether the output space is constrained (technical) or open (creative).
60 second bullets to scan on the way to the call.
When BLEU is a valid primary metric for translation (constrained output space)
When to push back on BLEU (creative content with broad valid output distributions)
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.