Eval Harness
Also known as: Eval framework
A framework for running, scoring, and tracking LLM evals over time; handles golden sets, judges, and regressions.
A reusable framework for running, scoring, and comparing LLM evaluations over time. Tracks model/prompt versions, golden-set results, regressions, and judge-LLM rubrics. Examples: OpenAI Evals, Inspect, Promptfoo, Braintrust.
In practice
Building one is a near-universal LLM-team task. Senior production interviews probe what belongs in a harness vs ad-hoc scripts.
How it compares
An eval harness is the framework that runs evals; a golden set is the input test data the harness consumes.
Related topics
Questions that mention this term
- Which of these are valid reasons NOT to fine-tune?
- You hear about 'eval harnesses' like Promptfoo and DeepEval. Explain what an eval harness does that a Jupyter notebook cannot.
- How do you detect…
- Design a RAG evaluation harness, what do you measure and how?
- Design eval infrastructure that makes a weekly model swap boring
- Select all components that are required to build a minimal end to end RAG system (the absolute basics, not nice to haves like rerankers or eval harnesses).
Related terms
Chatbot Arena
Crowdsourced blind A/B comparisons of LLM responses, aggregated into an Elo leaderboard.
Golden Set
Curated test cases with expected outputs; your regression suite for prompt and model changes.
Guardrails
Pre- and post-processing layers that block bad inputs/outputs and enforce policy on top of an LLM.
Hallucination
When a model confidently makes up something that isn't true.
HumanEval
Code-generation benchmark: 164 problems with hidden unit tests, scored by whether the generated code passes.
Jailbreak
Adversarial prompt that bypasses an LLM's safety training to elicit normally-refused content.