Keep the system prompt stable across all turns of a conversation. It is the standing-orders block: who the model is, what behavior it should exhibit, what constraints it should respect. That information does not change from turn to turn.
Per-turn dynamic data (the current user query, any retrieved context, transient state) belongs in the user message, not the system block. The user message is where the conversation actually changes; the system block is the stable scaffold around it.
There are two reasons to keep the system prompt stable. The first is provider caching. Anthropic, OpenAI, and most modern API providers cache stable prompt prefixes. If you keep the system block identical across turns, the provider's cache hits and you pay a fraction of the normal input cost (often 50 to 90 percent savings). The second reason is behavior consistency. A stable system block gives the model consistent standing orders; a varying system block produces inconsistent behavior across turns.
The only time to change the system prompt mid-conversation is on a fundamental mode shift, like pivoting from billing support to feature questions. That is a deliberate routing decision, not a per-turn habit.
**Stable system prompts** are the production default for multi-turn conversations. The system block carries standing orders (role, behavior, constraints); the user message carries per-turn dynamic data. Mixing those layers is a common but costly mistake.
The first reason for stability is prompt caching. Anthropic's prompt caching, OpenAI's automatic prefix caching, and Gemini's context caching all key on the prefix of the request. If the system block stays identical across turns, the prefix matches and the cache hits. Cached tokens are billed at a fraction of normal input cost (Anthropic at roughly 10 percent of base; OpenAI at 50 percent). Varying the system block per turn invalidates the cache every turn and pays the full input cost on a part of the request that has not changed.
The second reason is behavior consistency. The system block is the model's standing instruction set. Changing it every turn produces inconsistent behavior: the model might be terse on turn 3 and verbose on turn 4 because the system block subtly reframed its job. Production conversations rely on consistent behavior, and the cleanest path to that is keeping the system block stable.
Per-turn dynamic content (retrieved documents, current user query, transient state, tool results) belongs in the user message. That is where the conversation changes; that is where the cache misses are expected. Putting dynamic content in user messages keeps the system prefix cacheable and keeps behavior consistent.
The only valid reason to change the system prompt mid-conversation is a deliberate mode shift. The conversation pivots to a fundamentally different task and the model needs different standing orders. That is a routing decision, not a per-turn habit, and modern conversation designs handle it by starting a new session rather than mutating the existing one.
Quick reference
TL;DR
Keep the system prompt stable across turns to enable prompt caching and consistent behavior; put per-turn dynamic data in the user message, not the system block.
Think of the system prompt as the standing instructions you give to a new employee on day one: who they are, what they do, how they should behave. You do not rewrite the job description every time they answer a customer call. You let the standing instructions stay put and the customer-specific details show up in each conversation. Language models work the same way. The system message is the standing instructions; user messages are the per-call details. Rewriting the standing instructions every turn confuses the model and throws away a big cache discount the provider gives you for keeping prefixes stable.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Genuinely transient role mid-conversation If the conversation truly pivots to a different task category, mutating the system block is rarely the right move; start a new session with the new system block instead.
Per-conversation customization stitched into every system prompt If the customization changes per user, push it to the user message or to a clearly demarcated cache boundary so the rest of the prefix stays cacheable across users.
Debugging or A/B testing different system prompts Here you actively want the system prompt to vary across requests; just be aware that this is a development-mode pattern with full cache misses, not a production-mode pattern.
Quick scan · 2–4 min
Everything important, quickly.
3 min: the two reasons system stability matters (caching + behavior consistency), where dynamic data belongs, and how to handle mid-conversation mode shifts.
Real-world examples
Real products, models, and research that use this idea.
1Anthropic prompt caching offers about 90 percent off cached input tokens; production multi-turn routes lean on a stable system block to capture that discount.
2OpenAI automatic prefix caching applies 50 percent off matching prefixes; the design assumption is exactly that the system block does not change per turn.
3Production support agents structure the system block as stable role and policy text and put the customer-specific context in each user message.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
Sign in or go Pro to see the approach for this follow-up.
QWhen does a mid-conversation mode shift justify mutating the system block?
Sign in or go Pro to see the approach for this follow-up.
QHow would you organize a long system prompt to maximize cache hits across diverse conversations?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Editing the system prompt every turn with dynamic context, which invalidates the provider cache and produces inconsistent behavior across the conversation.