Why string-match accuracy fails as an LLM evaluation metric

Attempt it

Your answer

Quick reference

TL;DR

Open-ended outputs have many valid wordings and no single ground truth, quality is subjective and multi-dimensional, and the reliable signal, human grading, is slow and costly.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Imagine grading a class who all wrote essays answering 'why is the sky blue?'. There is no single correct sentence. One student writes a tidy paragraph, another a longer one, both are right. A grading machine that only checks 'does this match my one answer key word for word?' would fail almost everyone, even the good essays. That is what string-match does to language models: it has one reference answer and marks everything else wrong, even correct rephrasings. To grade fairly you need a thoughtful reader who weighs accuracy, clarity, and tone. But hiring human readers for thousands of essays is slow and expensive, which is exactly why grading open-ended answers is genuinely hard.

Key concepts

Eval fundamentals Ground truth LLM-as-judge

Detailed answer & concept explanation

~7 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Sign in for 5 free unlocks/day or go Pro for unlimited. See Pro pricing.

Quick scan · 2–4 min

Everything important, quickly.

3 min: no single ground truth, quality is multi-dimensional and subjective, human labels are expensive, so eval combines automated checks, an LLM judge, and human spot-checks.

Real-world examples

Real products, models, and research that use this idea.

Chatbot Arena ranks models by human pairwise votes precisely because no single reference answer exists for open-ended chat.
RAGAS scores faithfulness and relevance separately rather than one accuracy number, reflecting eval's multi-dimensional nature.
LangSmith and Promptfoo default to LLM-as-judge for open-ended outputs because string-match would mark valid paraphrases wrong.
HumanEval grades code by running tests, not by string-matching source, since many correct programs look completely different.
OpenAI Evals lets teams mix exact-match graders for structured tasks with model-graded checks for open-ended ones.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QIf string-match fails, why are BLEU and ROUGE still widely reported on generation tasks?

Sign in or go Pro to see the approach for this follow-up.

QHow would you make the expensive human-label problem tractable at scale?

Sign in or go Pro to see the approach for this follow-up.

QHow do you know your own human gold labels are trustworthy?

Sign in or go Pro to see the approach for this follow-up.

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Assuming a single reference answer plus exact match is enough. Open-ended outputs have many valid wordings, so string-match marks correct paraphrases wrong and reports a misleading score.

Sign in or go Pro to see all red flags and common mistakes. See Pro pricing.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.