When someone says 'eval prompt template,' what are they referring to and why does the wording matter so much?

Flashcard·Easy·4.0 · 0·~30s

Attempt it

An eval prompt template is the exact prompt sent to the LLM judge that frames the evaluation task: it includes the rubric, the model output to judge, optionally a reference answer, and instructions for how to score. Small wording changes (e.g., 'rate quality' vs 'rate helpfulness') can shift scores by 10 to 20 percent, making template design as important as the metric choice itself.

Your answer

When you use an LLM as a judge to evaluate another model's output, you need to tell the judge what to do. That instruction is the eval prompt template. An **eval prompt template** is a structured prompt that typically includes four components: the evaluation criteria or rubric (what dimensions to judge, what each score level means), the model output to evaluate, optionally a reference answer for comparison, and instructions for how to format the score. The template matters far more than most people expect. LLM judges are highly sensitive to prompt wording. Changing 'rate the quality of this response' to 'rate the helpfulness of this response' can shift the average score by 10 to 20 percent across the same set of outputs. That is not noise. It means the template is controlling what the judge pays attention to. This sensitivity has a practical consequence: the eval prompt template is a first-class artifact in the evaluation pipeline. It deserves the same iteration, version control, and testing as the production prompt. Teams that treat it as boilerplate end up with scores that reflect template wording rather than model quality.

An **eval prompt template** is the exact prompt sent to the LLM judge that defines the evaluation task. It contains the rubric (what to evaluate and what each score level means), the model output to judge, optionally a reference answer, and the expected output format (a score, a ranking, or structured JSON). The wording matters because LLM judges are highly sensitive to prompt phrasing. Research shows that swapping a single term, like 'quality' for 'helpfulness,' can shift average scores by 10 to 20 percent on the same outputs. Adding or removing anchor examples changes score distributions. Even the order in which components appear (rubric first vs output first) affects the judgment. This makes the eval prompt template one of the highest-leverage artifacts in the eval pipeline. Small template changes produce large score changes, which can flip model comparisons and deployment decisions. The practical discipline is to treat the template as a versioned, tested artifact. When you change it, re-run the eval on a calibration set and check whether judge-human agreement improved or degraded. In 2026 frameworks like Promptfoo, Braintrust, and LangSmith, the eval prompt template is an explicit, editable configuration object. Teams iterate on it the same way they iterate on production prompts: propose a change, test it, measure the effect, and commit.

Quick reference

TL;DR

An eval prompt template is the prompt sent to the LLM judge defining what to evaluate, how to score, and what criteria to apply. Small wording changes shift scores by 10 to 20 percent.

Memory aid

Easy to grasp

Imagine you are training a new teacher's assistant to grade essays. You hand them a sheet that says: 'Read the essay, check if it answers the question, and give it a score from 1 to 5. Here is what each score means.' That instruction sheet is the eval prompt template. Now imagine you change the sheet to say 'rate how creative the essay is' instead of 'rate how well it answers the question.' The assistant would grade the same essays completely differently. The template controls what the grader pays attention to. Even tiny changes in wording, like swapping 'quality' for 'helpfulness,' can change the scores by a lot, because the grader (the LLM judge) takes the instructions very literally.

Key concepts

Eval prompt design LLM-as-judge Rubric

Detailed answer & concept explanation

~4 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Quick scan · 2–4 min

Everything important, quickly.

5 min: define the eval prompt template, name its components, explain wording sensitivity, describe the calibration loop, and close with the practical discipline of treating it as a versioned artifact.

Real-world examples

Real products, models, and research that use this idea.

Promptfoo exposes the eval prompt template as an editable YAML field, letting teams customize criteria, anchor examples, and scoring instructions per evaluation dimension.
MT-Bench uses a carefully designed eval prompt template that instructs the judge to score on a 1 to 10 scale with explicit criteria for each range, plus chain-of-thought reasoning.
Braintrust AI lets teams A/B test eval prompt templates against human labels to find the template that produces the most reliable judge scores.
G-Eval introduced a template pattern where the judge generates chain-of-thought evaluation steps before assigning a score, which improved correlation with human judgments.
LangSmith provides default eval prompt templates for common dimensions (helpfulness, correctness, coherence) that teams customize for their domain.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QHow do you test whether a change to the eval prompt template actually improved judge accuracy?

QWhat is the role of chain-of-thought in eval prompt templates?

QHow do anchor examples in the rubric reduce score variance?

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Treating the eval prompt template as boilerplate. Small wording changes can shift judge scores by 10 to 20 percent, making it one of the highest-leverage artifacts in the eval pipeline.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

References

Primary sources. Skim if you want the original framing.