When someone says 'eval prompt template,' what are they referring to and why does the wording matter so much?

Flashcard·Easy·4.0 · 0·~30s·Asked atElevenlabs Spotify Voyage Ai·Relevant atAnthropic Anyscale Datadog

Attempt it

An eval prompt template is the exact prompt sent to the LLM judge that frames the evaluation task: it includes the rubric, the model output to judge, optionally a reference answer, and instructions for how to score. Small wording changes (e.g., 'rate quality' vs 'rate helpfulness') can shift scores by 10 to 20 percent, making template design as important as the metric choice itself.

Your answer

When you use an LLM as a judge to evaluate another model's output, you need to tell the judge what to do. That instruction is the eval prompt template. An **eval prompt template** is a structured prompt that typically includes four components: the evaluation criteria or rubric (what dimensions to judge, what each score level means), the model output to evaluate, optionally a reference answer for comparison, and instructions for how to format the score. The template matters far more than most people expect. LLM judges are highly sensitive to prompt wording. Changing 'rate the quality of this response' to 'rate the helpfulness of this response' can shift the average score by 10 to 20 percent across the same set of outputs. That is not noise. It means the template is controlling what the judge pays attention to. This sensitivity has a practical consequence: the eval prompt template is a first-class artifact in the evaluation pipeline. It deserves the same iteration, version control, and testing as the production prompt. Teams that treat it as boilerplate end up with scores that reflect template wording rather than model quality.

An **eval prompt template** is the exact prompt sent to the LLM judge that defines the evaluation task. It contains the rubric (what to evaluate and what each score level means), the model output to judge, optionally a reference answer, and the expected output format (a score, a ranking, or structured JSON). The wording matters because LLM judges are highly sensitive to prompt phrasing. Research shows that swapping a single term, like 'quality' for 'helpfulness,' can shift average scores by 10 to 20 percent on the same outputs. Adding or removing anchor examples changes score distributions. Even the order in which components appear (rubric first vs output first) affects the judgment. This makes the eval prompt template one of the highest-leverage artifacts in the eval pipeline. Small template changes produce large score changes, which can flip model comparisons and deployment decisions. The practical discipline is to treat the template as a versioned, tested artifact. When you change it, re-run the eval on a calibration set and check whether judge-human agreement improved or degraded. In 2026 frameworks like Promptfoo, Braintrust, and LangSmith, the eval prompt template is an explicit, editable configuration object. Teams iterate on it the same way they iterate on production prompts: propose a change, test it, measure the effect, and commit.

TL;DR

An eval prompt template is the prompt sent to the LLM judge defining what to evaluate, how to score, and what criteria to apply. Small wording changes shift scores by 10 to 20 percent.

Memory aid

Easy to grasp

Imagine you are training a new teacher's assistant to grade essays. You hand them a sheet that says: 'Read the essay, check if it answers the question, and give it a score from 1 to 5. Here is what each score means.' That instruction sheet is the eval prompt template. Now imagine you change the sheet to say 'rate how creative the essay is' instead of 'rate how well it answers the question.' The assistant would grade the same essays completely differently. The template controls what the grader pays attention to. Even tiny changes in wording, like swapping 'quality' for 'helpfulness,' can change the scores by a lot, because the grader (the LLM judge) takes the instructions very literally.

Key concepts

Eval prompt design LLM-as-judge Rubric

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

When teams adopt LLM-as-judge for evaluation, they quickly discover that the judge's scores depend not just on the outputs being evaluated but on how the judge is prompted. The eval prompt template is the artifact that controls this, and its design is one of the most consequential decisions in the eval pipeline.

This deep dive covers what an eval prompt template contains, why its wording affects scores so dramatically, and how to design and validate one that produces reliable judgments.

Anatomy of an eval prompt template

An eval prompt template is a structured prompt sent to the LLM judge. It typically contains four sections.

The rubric section defines what to evaluate. It names the quality dimensions (accuracy, helpfulness, tone, completeness) and specifies what each score level means. A well-designed rubric includes anchor examples: concrete outputs that illustrate what a 1, a 3, and a 5 look like. Without anchors, the judge interprets score levels abstractly and inconsistently.

The context section presents the original user prompt and the model's output. For reference-based evaluation, it also includes the gold answer. The order of these elements matters: some research suggests rubric-first produces more consistent scoring than output-first, because the judge reads the criteria before encountering the text.

The reasoning section instructs the judge to think step by step before assigning a score. This chain-of-thought pattern, popularized by G-Eval, improves judge consistency by forcing explicit consideration of each rubric dimension.

The output section specifies the format: a single integer, a JSON object with score and rationale, or a structured comparison result. Constraining the output format makes downstream parsing reliable.

Why wording sensitivity is so high

Designing a template that produces reliable scores

Validating the template against human labels

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Real products, models, and research that use this idea.

Promptfoo exposes the eval prompt template as an editable YAML field, letting teams customize criteria, anchor examples, and scoring instructions per evaluation dimension.
MT-Bench uses a carefully designed eval prompt template that instructs the judge to score on a 1 to 10 scale with explicit criteria for each range, plus chain-of-thought reasoning.

What an interviewer would ask next. Try answering before peeking at the approach.

QHow do you test whether a change to the eval prompt template actually improved judge accuracy?

Run both the old and new templates on a calibration set with human labels. Compare judge-human agreement (kappa or correlation) for each. If the new template produces higher agreement, the change improved accuracy. If not, revert.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Treating the eval prompt template as boilerplate. Small wording changes can shift judge scores by 10 to 20 percent, making it one of the highest-leverage artifacts in the eval pipeline.

60 second bullets to scan on the way to the call.

Define the eval prompt template as the prompt sent to the LLM judge
Name the four typical components: rubric, model output, optional reference, output format instructions

Primary sources. Browse if you want the original framing.