Select every artifact a prompt PR should produce in the CI run

Multi-select·Easy·4.0 · 0·~1 min·Asked atJasper Lightning Ai Roblox

Attempt it

A **prompt PR** is a change request to a prompt template, similar to a code PR. Eval-CI runs the prompt against a golden set of representative inputs and produces evidence the reviewer needs to decide whether to merge. Four artifacts make that decision possible. First, a **delta table** that shows every tracked metric (faithfulness, judge score, refusal rate, latency) for the new prompt next to the old one, with the percent change. Second, a **significance test** that says whether the change is statistically real or just sample noise. Third, a list of **worst-N failing examples**, with links to the actual inputs and outputs, so the reviewer can read them and spot regressions the aggregate metric missed. Fourth, a **cost-delta** based on the change in average token counts; a longer prompt costs more on every call forever, and the reviewer needs to see that. Two things are NOT artifacts of a prompt PR. A new **model checkpoint** is wrong because no training happens; only the prompt changed. **Automated approval** is wrong because eval-CI gates merge, it does not replace human review.

**Prompt PR eval-CI produces four artifacts: delta table, significance test, worst-N examples, cost-delta.** The two wrong options reveal two common confusions about what eval-CI is. The **delta table** is the at-a-glance summary. For each tracked metric (faithfulness, answer relevance, judge score, refusal rate), the table shows old, new, and percent delta. Reviewers scan this first. The **significance test** says whether the deltas are real or noise. With a golden set of even a few hundred examples, raw deltas of 1 to 3 points are usually inside the noise band; the significance test (paired bootstrap, McNemar, or similar) tells you whether to care. Without it, every merge becomes a coin flip on whether the prompt actually improved. The **worst-N failing examples** is where reviewers catch the regressions the aggregate metrics miss. A prompt can improve average judge score while wrecking one specific behavior (refusing to summarize legal documents, hallucinating on financial queries). Reading the worst 10 failures is the cheapest defense against that. The **cost-delta** catches the silent cost regression that an extra few-shot example or a longer system prompt introduces. A 200-token addition raises cost for every call forever; CI flags it now or finance flags it next month. The two wrong answers: a **model checkpoint** is wrong because prompt changes do not retrain anything (LLMOps Property: no retrain step). **Automated bypass of review** is the opposite of change management. Eval-CI gates merge by failing on regression; it does not replace the reviewer.

TL;DR

A prompt PR's CI run produces a delta table, a significance test, worst-case failing examples, and a cost-delta; it does not produce a model checkpoint and does not auto-approve.

Memory aid

Easy to grasp

Picture a chef tweaking a recipe and putting it up for a taste test. The kitchen produces a tasting card showing the new dish next to the old one (delta table), a statistician note saying whether the difference is real or just luck (significance test), the plates the diners liked least (worst-N examples), and an updated grocery bill (cost delta). The kitchen does not bake a new chef, and the head chef still has to sign off; nothing ships just because the tasting card looked good. That tasting card is what eval-CI hands the reviewer.

Key concepts

LLMOps

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Eval as CI is the LLMOps answer to 'how do I know this prompt change is safe to merge?' The mechanism mirrors a code PR: an automated job runs the change against a fixed test set, produces evidence, and posts that evidence into the PR for a human reviewer. The four artifacts that matter on the output side are a delta table, a significance test, a worst-N failing-examples list, and a cost-delta. The two artifacts on the wrong-answer list (model checkpoint, automated approval) reveal two of the most common confusions about what eval-CI is and what it is not.

This walkthrough breaks down each artifact, why it is necessary, the production shape of a typical eval-CI job, and how the gate interacts with human review.

Mental model: eval-CI produces an evidence package; the human reviewer makes the merge decision. The gate's job is to block obviously regressive merges and surface the data the reviewer needs for everything else.

The delta table and the significance test

Delta table

The delta table is the at-a-glance summary every reviewer reads first. Rows are metrics: faithfulness, answer relevance, context precision (for RAG), pairwise judge wins, refusal rate, toxicity rate, latency p50 and p99. Columns are old prompt, new prompt, absolute delta, percent delta. The table renders as Markdown in the PR comment, with red and green highlighting for regressions and improvements.

The metrics chosen reflect the product: a RAG assistant tracks groundedness and citation precision; a chatbot tracks judge wins on a held-out dialog set; an agent tracks tool-use correctness. The set is fixed for the lifetime of the prompt family so deltas are comparable across PRs.

Significance test

A delta on its own is meaningless without an error bar. A golden set of 500 examples and a judge with measurable variance can easily produce 2-point swings in either direction from pure noise. The significance test answers 'is this delta real?'

Three common choices:

Paired bootstrap on per-example score deltas. Resample with replacement, compute the delta on each resample, report the 95 percent CI of the delta.
Wilcoxon signed-rank when the metric is ordinal and the assumption of normality is shaky.
McNemar's test for binary outcomes (pass/fail, refusal/not).

The artifact in the PR comment is a CI or p-value next to each metric in the delta table. Reviewers learn to ignore deltas inside the CI and act on deltas outside it. Without this discipline, the team chases noise and the gate loses credibility.

Worst-N failing examples and cost-delta

Why the wrong answers are wrong

Production shape of an eval-CI job

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Real products, models, and research that use this idea.

Promptfoo's GitHub Action posts a delta table and PR comment by default and is widely used as the eval-CI runner.
DeepEval (Confident AI) ships a similar pattern with pytest-style test cases that fail on regression beyond thresholds.

What an interviewer would ask next. Try answering before peeking at the approach.

QHow would you design the worst-N failing examples view so reviewers actually look at it?

Surface each failure with the input, both outputs side by side, the judge score for each, and a one-line diff hint. Link to the full trace in LangSmith or Langfuse. Cap at 10 to 20 examples; longer lists get skipped.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Treating eval-CI as automated merge. CI gives the reviewer evidence; the human still approves the change, especially when the deltas are small.

60 second bullets to scan on the way to the call.

The four artifacts an eval-CI run should produce on a prompt PR
Why a model checkpoint is not an eval-CI artifact

Primary sources. Browse if you want the original framing.