You're an AI engineer running a production LLM app for medical-document summarization on Claude Sonnet 4.x. The current prompt is zero-shot. A team member proposes adding few-shot examples to improve quality. Design the experiment (golden set, metrics, statistical analysis, decision criteria) that would let you confidently say 'few-shot beats zero-shot for this task on this model' or 'zero-shot is fine, save the tokens'. Be specific about what makes the experiment statistically sound.
Free · 2 AI evals / day
The experiment needs to answer a simple yes-or-no question with enough rigor that you would bet a production rollout on it. That means a real golden set, real metrics, and real statistics.
Start with the data. Build a golden set of 100-300 (input, ideal-output) pairs that covers representative cases, edge cases, and recent production failures. Hold out 20-30% as a final test set that you do not use during iteration.
Then pick the metrics. Use two or three that capture different aspects of quality. For medical-document summarization, that usually means faithfulness (does the summary contradict the source), coverage (does it include the key facts), and format compliance (does it match the required structure). Single metrics hide what is improving.
Run both prompts on the same model with the same temperature and same retrieval setup. Score each output on each metric using LLM-as-judge or a rubric, not vibes. Then compute confidence intervals (bootstrap is the standard) so you know whether the difference is real or noise.
Finally, write down the win threshold before you see the results, and add a cost column. A small quality win at double the token cost may not be worth shipping.
Seven steps make this experiment trustworthy.
**Step one: golden set.** 100-300 (input, ideal-output) pairs. Cover representative inputs, edge cases, and recent production failures. For medical-document summarization, domain experts must label. Lay labels miss the failure modes that matter. Hold out 20-30% as a final test set.
**Step two: variants.** Write a zero-shot prompt and a few-shot prompt with 3-5 examples. Pick few-shot examples for diversity, not for similarity to the golden set. Cherry-picking matches the eval distribution and inflates the lift.
**Step three: metrics.** Two or three orthogonal scores. Faithfulness (contradiction-free), coverage (key facts present), format compliance (structure matches). Single-metric experiments hide tradeoffs.
Step four is the run itself. Same Claude Sonnet 4.x version, same temperature, same retrieval. Score with LLM-as-judge or rubric, not vibes.
Step five is statistics. Bootstrap 1000+ resamples for 95% confidence intervals on each metric. A few-shot win is real when the CIs do not overlap and the mean delta exceeds the noise floor.
Step six: pre-register the decision threshold. Write the win criterion before you see the results. Example: faithfulness up at least 5 points with CI non-overlap and format compliance up at least 5 points.
**Step seven: cost accounting.** Measure token cost per query. A 3-point lift at 2x cost is usually not worth shipping.
Pitfalls: small N, no edge cases, over-fit example selection, single metric, no significance test, no pre-registered threshold.
Quick reference
TL;DR
Build a 100-300 pair golden set with held-out test, score on 2-3 orthogonal metrics with bootstrap confidence intervals, pre-register the win threshold, and weigh the lift against the token-cost delta.
Imagine you want to know whether a new recipe is actually better than your old one. You do not just cook each version once for your family and see who smiles more. You cook both versions for fifty different people, you score on three dimensions (taste, appearance, how full they felt), you decide in advance that better means at least four points higher on taste, and you check whether the gap is bigger than the normal day-to-day variation in how good your cooking is. Then you also count what each version cost in groceries. Only when the new recipe wins on taste by a clear margin and the extra cost is worth it do you say it is actually better. The prompt experiment works the same way.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
When you cannot get domain experts to label the golden set for medical content Lay labels miss the clinical-accuracy failure modes that matter; running the experiment against noisy labels produces a confident but wrong answer.
When the production traffic shape is not represented in the golden set Wins on representative cases can mask regressions on long-tail inputs that drive incident reports; bias the eval toward the actual traffic mix.
When you do not have budget for at least 100 paired evaluations per variant Bootstrap CIs explode below that floor; the experiment cannot distinguish real lift from noise and any answer is over-claiming certainty.
When you cannot define the cost-quality decision rule in advance Without the pre-registered threshold, the team will rationalize whichever variant looks better and the experiment loses its purpose.
Quick scan · 2–4 min
Everything important, quickly.
5 min: golden-set construction + 2-3 orthogonal metrics + bootstrap CIs + pre-registered threshold + cost accounting + common pitfalls (small N, cherry-picked examples, single metric, post-hoc threshold).
Real-world examples
Real products, models, and research that use this idea.
1Anthropic's prompt-engineering documentation explicitly recommends evaluating Claude prompts on held-out sets with multiple metrics rather than single-shot vibe checks.
2OpenAI Evals ships a bootstrap confidence interval scorer and a templated rubric harness so prompt teams can compare zero-shot and few-shot variants without writing the statistics themselves.
3Promptfoo and Braintrust both expose pre-registered decision thresholds in their A/B test runners, treating threshold setting as a first-class step in the experiment.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you choose between bootstrap CIs and a paired t-test for this comparison?
Sign in or go Pro to see the approach for this follow-up.
QHow would you stress-test the few-shot variant against adversarial inputs not in the golden set?
Sign in or go Pro to see the approach for this follow-up.
QWhen would you skip the experiment and ship the zero-shot version directly?
Sign in or go Pro to see the approach for this follow-up.
QHow does this experiment design change when temperature is non-zero?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Declaring a few-shot win after running ten examples and looking at point-estimate accuracy, ignoring noise floor, multi-dimension metrics, and the token-cost delta.