Eval Harness
Also known as: Eval framework
A framework for running, scoring, and tracking LLM evals over time; handles golden sets, judges, and regressions.
A reusable framework for running, scoring, and comparing LLM evaluations over time. Tracks model/prompt versions, golden-set results, regressions, and judge-LLM rubrics. Examples: OpenAI Evals, Inspect, Promptfoo, Braintrust.
In practice
Building one is a near-universal LLM-team task. Senior production interviews probe what belongs in a harness vs ad-hoc scripts.
How it compares
An eval harness is the framework that runs evals; a golden set is the input test data the harness consumes.
Related topics
Related terms
Hallucination
When a model confidently makes up something that isn't true.
LLM Evaluation
Measuring whether an LLM does what you want, beyond "looks fine to me".
Perplexity
Exp(average cross-entropy) on held-out text; lower means the model is less surprised by real data.
Reasoning Model
An LLM trained to reason at length internally before answering. Slower and more expensive, but much better on hard problems.
Guardrails
Pre- and post-processing layers that block bad inputs/outputs and enforce policy on top of an LLM.
MMLU (Massive Multitask Language Understanding)
Multiple-choice benchmark across 57 academic subjects; the standard "raw knowledge" headline number.