Why string match accuracy fails as an LLM evaluation metric
Open-ended outputs have many valid wordings and no single ground truth, quality is subjective and multi-dimensional, and the reliable signal, human grading, is slow and costly.
Imagine grading a class who all wrote essays answering 'why is the sky blue?'. There is no single correct sentence. One student writes a tidy paragraph, another a longer one, both are right. A grading machine that only checks 'does this match my one answer key word for word?' would fail almost everyone, even the good essays. That is what string-match does to language models: it has one reference answer and marks everything else wrong, even correct rephrasings. To grade fairly you need a thoughtful reader who weighs accuracy, clarity, and tone. But hiring human readers for thousands of essays is slow and expensive, which is exactly why grading open-ended answers is genuinely hard.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
LLM evaluation is hard for a reason that sounds simple but runs deep: for most things we ask a model to generate, there is no single correct answer to match against. Summaries, explanations, emails, and chat replies all have a large space of valid forms. The instinct to measure accuracy with exact string-match, which works perfectly for classification or arithmetic, falls apart the moment outputs are open-ended.
The contrast with classic machine learning is sharp. An image classifier predicts one of a fixed set of labels, so accuracy is a clean count of right versus wrong. A regression model has a numeric target, so error is just a distance. Generative language has neither a fixed label set nor a single target string, so the comfortable metrics from those settings do not carry over. This is not a tooling gap that better libraries will close; it is a property of the task itself.
This deep dive unpacks the three forces that make open-ended evaluation difficult: the absence of a single ground truth, the multi-dimensional and subjective nature of quality, and the cost of the only fully reliable signal, human judgment. Then it covers how modern eval stacks work around these forces by layering complementary methods, and when the difficulty simply does not apply.
No single ground truth: the many valid answers problem
Ask ten competent people to summarise the same article and you get ten different summaries, all acceptable. The same is true for explanations, translations, and chat replies. The space of correct outputs is large and the wordings vary wildly. That is the defining feature of open-ended generation, and it is exactly what string-match cannot model.
Exact string-match assumes the opposite. It holds one reference string and asks whether the candidate matches it character for character. A correct paraphrase using synonyms or a different sentence order scores zero. The dataset author picked one phrasing out of thousands, and that arbitrary choice silently becomes the only answer the metric will accept. So the metric is wrong in both directions: it punishes valid answers, and on tasks where the model memorised the reference it can reward shallow recall over genuine understanding.
You might think adding a handful of reference answers fixes this. It does not. The number of valid phrasings for even a short factual question grows faster than any reference list you would maintain by hand, and for a multi-sentence explanation it is effectively unbounded. Enumerating the correct set is not a practical strategy; you have to measure meaning instead.
This is why a high string-match score can mask a weak model and a low score can hide a strong one. The number simply does not track the thing you care about, which is whether the answer is good. Worse, the failure is silent: the metric returns a confident percentage that looks rigorous on a dashboard while quietly measuring the wrong thing.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Chatbot Arena ranks models by human pairwise votes precisely because no single reference answer exists for open-ended chat.
- RAGAS scores faithfulness and relevance separately rather than one accuracy number, reflecting eval's multi-dimensional nature.
What an interviewer would ask next. Try answering before peeking at the approach.
QIf string-match fails, why are BLEU and ROUGE still widely reported on generation tasks?
Discuss n-gram overlap as a cheap proxy, its weak correlation with human judgment on open-ended text, and why teams still report it for comparability while supplementing with judge or human eval.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Assuming a single reference answer plus exact match is enough. Open-ended outputs have many valid wordings, so string-match marks correct paraphrases wrong and reports a misleading score.
60 second bullets to scan on the way to the call.
Why exact string-match breaks on open-ended outputs
What having no single ground truth means for scoring
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.