You're building a medical document Q&A system: clinicians ask questions, your app retrieves relevant patient note chunks, and the LLM answers based on those chunks. Specify the structural layout of the prompt: what each component is, where it lives (system vs user), and why. Address: instructions, output format, few-shot examples, refusal directive, citation constraint, context layout. Be specific about decisions and tradeoffs.
Put role, format schema, refusal rule, citation contract, and few-shot examples in the cacheable system message; put XML-tagged retrieved notes and the clinician's question, in that order, in the user message.
Imagine you are a brand-new hospital intern, and your supervisor hands you a binder of rules every morning, then on each patient case slides a stack of that patient's notes across your desk and asks one question. The binder never changes day to day. It tells you to answer only from the notes, write your answer in a fixed form, and admit when the notes do not say enough. The stack of notes is different every case, and the question always comes after the notes so it is the last thing you read before you answer. That two-part split (rules in the binder, evidence and question on the desk) is exactly how this prompt is laid out.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Medical-document Q&A is the kind of task where prompt design either makes or breaks the product. The stakes are high (hallucinated medical claims are unacceptable), the inputs are messy (notes from many encounters, different formats, partial coverage), and the downstream consumers are humans who will act on the answer. Every structural choice in the prompt has to be defended.
This question forces the candidate to commit to a layout, name each component, and explain the tradeoffs. The right answer is not a list. It is a layered argument: which parts are stable and cacheable, which parts vary per call, where attention is strongest, what makes citations checkable, and what happens when the notes do not contain the answer.
This deep dive walks through the layered argument. By the end you should be able to draw the prompt on a whiteboard, defend each placement decision, and name the production tooling that backs the design.
Why split system from user at all
The system vs user split is a human convention with three real production payoffs.
The first is prompt caching. Hosted providers (Anthropic Claude Opus 4.7, OpenAI GPT-5.5) let you mark a cache breakpoint after the system block. Everything before the breakpoint is reused across calls; only the dynamic tail is re-processed. For a medical Q&A prompt with two or three few-shot examples, the system block can be 4 to 8k tokens. Caching that turns the per-call cost into mostly the user-side payload, which is typically the notes plus the question. The savings are 50 to 90 percent depending on traffic patterns.
The second is attention placement. Long-context LLMs attend most strongly to the start and end of the input. The system block sits at the start; the question sits at the end of the user message. Both ends get the strongest attention. The retrieved notes, which sit in the middle of the user message, get the weakest attention, which is exactly where you want the bulk of the evidence to live because the model will be pulled toward it by the question at the tail.
The third is eval discipline. The system block is what you author and version. It runs through regression CI on every change. The user block is what flows in at request time; it runs through retrieval-side monitoring. Different processes, different tooling, different alerts. Keeping the split clean keeps the engineering organization clean.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Anthropic Claude Opus 4.7 with prompt caching is widely used for clinical decision support pilots where the stable system block (schema + few-shot) is cached and only the patient notes and question vary per call.
- OpenAI GPT-5.5 in JSON-mode is used by RAG vendors to enforce structured outputs with explicit confidence fields, with downstream code routing low-confidence answers to human review.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you regression-test this prompt before shipping a change?
Golden set of (notes, question, ideal JSON) triples; metrics for faithfulness (verbatim citation match), format compliance, and refusal correctness; block merge on regressions past noise floor.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Putting retrieved patient notes in the system message because they look like context, which breaks prompt caching and loses the recency-attention boost that the clinician's question needs.
60 second bullets to scan on the way to the call.
The system vs user boundary and why it matches caching
The output schema with confidence enum
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.