You're an AI engineer running a production LLM app for medical document summarization on Claude Sonnet 4.x. The current prompt is zero-shot. A team member proposes adding few-shot examples to improve quality. Design the experiment (golden set, metrics, statistical analysis, decision criteria) that would let you confidently say 'few-shot beats zero-shot for this task on this model' or 'zero-shot is fine, save the tokens'. Be specific about what makes the experiment statistically sound.
Build a 100-300 pair golden set with held-out test, score on 2-3 orthogonal metrics with bootstrap confidence intervals, pre-register the win threshold, and weigh the lift against the token-cost delta.
Imagine you want to know whether a new recipe is actually better than your old one. You do not just cook each version once for your family and see who smiles more. You cook both versions for fifty different people, you score on three dimensions (taste, appearance, how full they felt), you decide in advance that better means at least four points higher on taste, and you check whether the gap is bigger than the normal day-to-day variation in how good your cooking is. Then you also count what each version cost in groceries. Only when the new recipe wins on taste by a clear margin and the extra cost is worth it do you say it is actually better. The prompt experiment works the same way.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
This question is about epistemic discipline, not about prompt engineering. Plenty of teams have shipped prompt changes based on twenty examples and a feeling that the new one was better. Some of those changes regressed production quality and nobody noticed for weeks because the eval was not designed to catch the regression in the first place.
The medical-document setting raises the stakes. A faithfulness regression in a clinical-summary pipeline can route the wrong information to a downstream system, and the cost of that error is real. The experiment design has to be careful enough that you would defend it under audit, not just careful enough that you feel good shipping the change.
The deep dive walks through what makes the experiment trustworthy: the golden set, the metric design, the statistical analysis, the pre-registered decision rule, and the cost accounting that turns the result into a ship or do-not-ship call.
Golden set construction
The golden set is the foundation; everything else fails if this is wrong. The target size is 100-300 (input, ideal-output) pairs. Below 100 the bootstrap confidence intervals are too wide to distinguish small effects from noise. Above 300 the labeling cost compounds without much added power; the marginal return on each extra pair is small once you are past 200.
Composition matters as much as size. Three slices belong in the set. The first is representative inputs sampled from production traffic, weighted by frequency. The second is edge cases that the current prompt has been observed to fail on, pulled from incident reports and customer complaints. The third is recent production failures from the last 4-8 weeks, which captures distribution drift and emerging failure modes. A set that is only representative misses the long tail; a set that is only edge cases biases the eval toward worst-case behavior.
Labeling requires domain experts for medical content. Clinical accuracy, terminology, and contradiction detection all require someone who has seen the right answers. Lay labels look fine on the surface and quietly miss the failures that matter. Plan for inter-annotator agreement spot checks (10-20% of the set double-labeled) so you can quantify label noise; bootstrap CIs that ignore label noise underestimate uncertainty.
Finally, hold out 20-30% as a final test set. You touch this set only at the end, to score the final variants. The remaining 70-80% is the iteration set, used for prompt design, example selection, and intermediate evaluation. Iterating against the test set is the most common way to over-fit a prompt experiment.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Anthropic's prompt-engineering documentation explicitly recommends evaluating Claude prompts on held-out sets with multiple metrics rather than single-shot vibe checks.
- OpenAI Evals ships a bootstrap confidence interval scorer and a templated rubric harness so prompt teams can compare zero-shot and few-shot variants without writing the statistics themselves.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you choose between bootstrap CIs and a paired t-test for this comparison?
Paired t-test assumes normality on the per-example deltas; bootstrap makes no parametric assumption and handles skewed metric distributions; default to bootstrap unless you have reason to trust normality.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Declaring a few-shot win after running ten examples and looking at point-estimate accuracy, ignoring noise floor, multi-dimension metrics, and the token-cost delta.
60 second bullets to scan on the way to the call.
Golden-set size and composition rules
Why 2-3 orthogonal metrics beats a single conflated score
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.