The four correct answers are role definition, output-format directive, refusal directive, and tone/style. The two wrong answers are the user's current question and retrieved RAG context.
The rule is simple: **persistent vs per-turn**. The system message is the model's standing orders. It carries things that should be true on every call in the session: who the model is, how it should answer, what format to use, what it must refuse. You set it once and forget about it.
The user message is what changes each time. The current question is obviously per-turn. Retrieved RAG context is per-turn too, even though it can feel like 'background': the retriever pulls different chunks for every query. If you put those chunks in system, you would have to rewrite the system block on every call, which defeats the purpose of having a stable system.
When in doubt, ask: does this content survive to the next call unchanged? If yes, system. If no, user.
Four belong in system: role definition, output-format directive, refusal-behavior directive, and tone/style. They are persistent rules the model should follow across every turn in this session, set once and reused.
Two belong in user. The user's current question is the per-turn input; it has no other home. Retrieved RAG context goes in user too, and this trips people up. The chunks change every query, so they cannot be cached as a stable system prefix. Modern guidance from both Anthropic and OpenAI recommends placing retrieval results close to the query to win the recency-attention bias.
The discipline that follows: keep the system block stable and big, put dynamic per-turn data in user. In 2026 this also has a direct cost consequence. Prompt caching gives a stable system prefix roughly a tenth of normal input cost on cache hits. Every dynamic byte you put in system is a byte that breaks the cache and pays full price.
Quick reference
TL;DR
Anything stable across the session belongs in system; anything that changes per turn (the user's question, retrieved RAG chunks) belongs in user.
Picture two folders on a clipboard for a help-desk worker. The top folder is the rulebook: who they work for, how to talk, what to refuse, what format their replies take. They open it once a day and follow it on every call. The bottom folder gets a new sheet of paper every time the phone rings: today's caller name, today's question, the printout the system just pulled about that caller's account. The rulebook is the system message. The fresh sheets are the user message. Mixing them, taping today's caller note inside the rulebook: breaks the system the next time the phone rings.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
A pipeline that uses only the legacy completions endpoint without roles There is no system/user split; you concatenate everything into one prompt with delimiters. The placement question becomes 'where in the string', not 'which role'.
Very short single-turn calls where caching cannot pay off If you only ever make one call per session, the cache-hit argument disappears. The role split still helps the model attend correctly, but it is no longer a cost lever.
Heavily personalized system prompts that change per user with no cache-control breakpoints Without explicit cache breakpoints, per-user data in system invalidates the whole prefix; you may be better off keeping system static and putting personalization in user.
Quick scan · 2–4 min
Everything important, quickly.
3 min: persistent vs per-turn rule + where RAG context goes + recency attention + prompt-caching cost + one production failure mode.
Real-world examples
Real products, models, and research that use this idea.
1Anthropic's prompt-engineering guide explicitly recommends user-message placement for retrieved RAG chunks; Claude Opus 4.7 attends better to context placed close to the query.
2OpenAI's structured-outputs and JSON-mode guidance puts the schema directive in system (stable) and the data to extract in user (dynamic).
3Cursor keeps editor-wide coding conventions in system and rebuilds the user message every keystroke with the active file, cursor span, and retrieved related files.
4Customer-support copilots like Intercom Fin lock brand persona and refusal rules in system, then resend the live ticket and retrieved knowledge-base snippets in user on every turn.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QSuppose you must personalize the system block per user. How do you preserve a high cache hit rate?
Sign in or go Pro to see the approach for this follow-up.
QIf you put RAG chunks in system anyway, what specifically breaks first at scale?
Sign in or go Pro to see the approach for this follow-up.
QHow would you place tool definitions (function-calling JSON schemas): system or user?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Putting retrieved RAG chunks into the system message because they 'feel like context'; they change every query, so they belong in user near the question.