For a machine translation eval, a senior engineer says BLEU is fine as the primary metric. When are they right, and when should you push back? · Zenaique

Attempt it

Quick reference

TL;DR

BLEU works for constrained translation (technical, legal) with high-quality references. Push back for creative content where valid translations diverge, and add COMET or BERTScore for semantic equivalence.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Imagine translating an instruction manual versus translating a poem. The manual has one right way to say each step, so checking word overlap with the answer key works. The poem has a hundred ways to express the same feeling, so checking word overlap penalizes beautiful translations that use different words. BLEU is the word-overlap checker. It works for the manual but fails for the poem.

Key concepts

BLEU Machine Translation BERTScore

Detailed answer & concept explanation

~3 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Sign in for 5 free unlocks/day or go Pro for unlimited. See Pro pricing.

Quick scan · 2–4 min

Everything important, quickly.

Open with the output-entropy framing: constrained tasks have tight valid-output distributions where BLEU works; creative tasks have broad distributions where it fails. Walk when the senior engineer is right (technical, legal) and when to push back (marketing, dialog). Introduce COMET and BERTScore as complements. Close with the production stack: BLEU for fast CI, COMET for quality, human eval for calibration.

Real-world examples

Real products, models, and research that use this idea.

WMT (Workshop on Machine Translation) reports BLEU alongside COMET and human evaluation scores, recognizing that BLEU alone is insufficient for creative content.
Google Translate's internal evaluation uses BLEU for regression detection across all language pairs and COMET for quality assessment on creative and marketing content.
COMET was developed by Unbabel and trained on millions of human quality annotations, making it the strongest learned metric for translation evaluation as of 2026.
DeepL uses a combination of automated metrics and human post-editing rates, with BLEU serving as a fast CI check rather than the primary quality signal.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QCOMET is trained on human judgments. What happens when the human judgment distribution shifts (different annotator pool, different quality standards)?

Sign in or go Pro to see the approach for this follow-up.

QYou add COMET to the stack and it disagrees with BLEU on 20% of translations. How do you decide which metric is right?

Sign in or go Pro to see the approach for this follow-up.

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Applying BLEU uniformly across all translation tasks without considering whether the output space is constrained (technical) or open (creative).

Sign in or go Pro to see all red flags and common mistakes. See Pro pricing.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.