Pick the best definition: in LLM evaluation, a metric is _____.

A **metric** in LLM evaluation is a function that takes model output and returns a number. That number is what lets you compare quality across runs, models, and time. Some metrics are reference-based: they compare the output to a known good answer. BLEU compares n-gram overlap with a reference translation. ROUGE compares n-gram overlap with a reference summary. F1 compares predicted entities against gold-labeled entities. These work well when you have a clear expected answer. Other metrics are reference-free: they score the output on its own. Perplexity measures how surprised the model is by the text. LLM-judge scores rate the output on dimensions like helpfulness or accuracy using a rubric, but the final number is still a metric. Faithfulness scores in RAG evaluation check whether the answer is grounded in the retrieved context. The reason metrics matter is comparability. If you change a prompt and rerun the model, the metric tells you whether the change helped, hurt, or made no difference. Without a metric, you are reading outputs and guessing. Choosing the wrong metric is the most common mistake. BLEU is great for translation but terrible for open-ended generation. Perplexity is great for language modeling but tells you nothing about instruction following.

In LLM evaluation, a **metric** is a quantitative function that takes model output (and optionally a reference) and returns a numeric score. The score makes quality measurable and comparable across runs, models, and time. Metrics divide into two families. Reference-based metrics compare the output to a known correct answer. BLEU measures n-gram precision against reference translations. ROUGE measures n-gram recall against reference summaries. F1 computes the harmonic mean of precision and recall for classification and extraction tasks. `pass@k` checks whether at least one of k code samples passes a test suite. Reference-free metrics score the output without a gold answer. Perplexity measures the model's surprise on held-out text. LLM-as-judge scores apply a rubric via a strong model and return a numeric rating. BERTScore computes semantic similarity using contextual embeddings rather than exact n-gram overlap. Choosing the right metric is the single most consequential decision in eval design. The metric defines what 'better' means. If you pick BLEU for open-ended generation, you will optimize for surface overlap with a reference, which penalizes valid paraphrases. If you pick perplexity for an instruction-following task, you will optimize for fluent language modeling, which says nothing about whether the model actually followed the instruction. The practical rule: pick a metric that aligns with what your users care about, validate it against human judgment on a small sample, and treat disagreements between the metric and human raters as a signal to revisit your choice.