Why string-match accuracy fails as an LLM evaluation metric

Evaluating a language model is hard because there is usually no single right answer. Ask a model to summarise an article or explain a concept, and there are dozens of correct ways to do it. A grading method that checks for an exact match against one reference answer will mark almost every valid response wrong. **String-match** accuracy works fine when there is one correct token, like a math result or a yes/no label. It breaks completely on open-ended text. A correct paraphrase that uses different words scores zero, so the number you get does not reflect real quality. Quality itself is also subjective and multi-dimensional. A good answer must be accurate, relevant, complete, and well-toned all at once. One pass/fail string comparison cannot capture any of that. That is why real evaluation leans on humans or on another model acting as a judge. Both can read meaning rather than match letters. The catch is that human labels are slow and expensive, which is the core difficulty of the whole field.

Evaluating open-ended LLM outputs is hard for three linked reasons. First, there is no single ground truth. A summary or explanation has many valid wordings. Exact string-match marks every correct paraphrase as wrong, so the score misleads in both directions. Second, quality is subjective and multi-dimensional. A good answer must be faithful, relevant, complete, and appropriately toned. One binary match collapses those into a single pass/fail signal that captures none of them well. Third, the reliable fallback is expensive. Humans can read meaning and judge nuance, but labelling thousands of outputs is slow and costly. That bottleneck is the real pain point. The practical answer is to combine three modes. Use automated checks where output is structured. Use an LLM as judge for things that need general intelligence but not deep expertise. Reserve human review for high-stakes spot checks. This is the standard pattern in 2026 eval stacks.