Pick the best definition: in LLM evaluation, a metric is _____.

Attempt it

A **metric** in LLM evaluation is a function that takes model output and returns a number. That number is what lets you compare quality across runs, models, and time. Some metrics are reference-based: they compare the output to a known good answer. BLEU compares n-gram overlap with a reference translation. ROUGE compares n-gram overlap with a reference summary. F1 compares predicted entities against gold-labeled entities. These work well when you have a clear expected answer. Other metrics are reference-free: they score the output on its own. Perplexity measures how surprised the model is by the text. LLM-judge scores rate the output on dimensions like helpfulness or accuracy using a rubric, but the final number is still a metric. Faithfulness scores in RAG evaluation check whether the answer is grounded in the retrieved context. The reason metrics matter is comparability. If you change a prompt and rerun the model, the metric tells you whether the change helped, hurt, or made no difference. Without a metric, you are reading outputs and guessing. Choosing the wrong metric is the most common mistake. BLEU is great for translation but terrible for open-ended generation. Perplexity is great for language modeling but tells you nothing about instruction following.

In LLM evaluation, a **metric** is a quantitative function that takes model output (and optionally a reference) and returns a numeric score. The score makes quality measurable and comparable across runs, models, and time. Metrics divide into two families. Reference-based metrics compare the output to a known correct answer. BLEU measures n-gram precision against reference translations. ROUGE measures n-gram recall against reference summaries. F1 computes the harmonic mean of precision and recall for classification and extraction tasks. `pass@k` checks whether at least one of k code samples passes a test suite. Reference-free metrics score the output without a gold answer. Perplexity measures the model's surprise on held-out text. LLM-as-judge scores apply a rubric via a strong model and return a numeric rating. BERTScore computes semantic similarity using contextual embeddings rather than exact n-gram overlap. Choosing the right metric is the single most consequential decision in eval design. The metric defines what 'better' means. If you pick BLEU for open-ended generation, you will optimize for surface overlap with a reference, which penalizes valid paraphrases. If you pick perplexity for an instruction-following task, you will optimize for fluent language modeling, which says nothing about whether the model actually followed the instruction. The practical rule: pick a metric that aligns with what your users care about, validate it against human judgment on a small sample, and treat disagreements between the metric and human raters as a signal to revisit your choice.

Quick reference

TL;DR

A metric is a quantitative function that scores model output, making quality measurable and comparable across runs, models, and time.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Imagine you are grading essays. You could write comments on each one (that is a rubric), or you could assign a number out of 100 (that is a metric). The number is what lets you compare: you can say this essay scored 82 and that one scored 74, so this one is better on whatever you measured. In LLM evaluation, a metric does the same thing. BLEU counts how many word sequences match a reference translation and gives you a number between 0 and 1. ROUGE counts how many reference word sequences appear in the summary and gives you a number. F1 balances false positives and false negatives into one score. The point is that a metric always produces a number. Without numbers, you cannot track improvement, detect regressions, or compare models. Choosing the wrong metric is one of the most common eval mistakes because you end up optimizing for the wrong thing.

Key concepts

Metric Quantitative Scoring Reference-Based Reference-Free Comparability

Detailed answer & concept explanation

~5 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Sign in for 5 free unlocks/day or go Pro for unlimited. See Pro pricing.

Quick scan · 2–4 min

Everything important, quickly.

5 min: define metric as a quantitative scoring function, distinguish reference-based from reference-free with examples, explain why metric selection determines what you optimize, give a concrete misapplication example, and close with the practice of layering multiple metrics.

Real-world examples

Real products, models, and research that use this idea.

Google uses BLEU as a primary metric for its translation systems because translation has constrained valid outputs where n-gram precision tracks human preference well.
Anthropic evaluates Claude's code generation using pass@k on HumanEval and SWE-bench, where functional correctness is the ground truth rather than textual similarity.
Ragas, an open-source RAG evaluation framework, defines metrics like faithfulness and context recall that check whether retrieved context actually supports the generated answer.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QHow do you validate that your chosen metric actually correlates with user-perceived quality?

Sign in or go Pro to see the approach for this follow-up.

QWhat is BERTScore and when does it outperform BLEU and ROUGE?

Sign in or go Pro to see the approach for this follow-up.

QWhen two metrics disagree about which model is better, how do you resolve the conflict?

Sign in or go Pro to see the approach for this follow-up.

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Confusing metrics with rubrics. A rubric defines what good and bad look like; a metric produces a number. You need both, but they serve different purposes.

Sign in or go Pro to see all red flags and common mistakes. See Pro pricing.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.