Keep the system prompt stable across turns to enable prompt caching and consistent behavior; put per-turn dynamic data in the user message, not the system block.
Think of the system prompt as the standing instructions you give to a new employee on day one: who they are, what they do, how they should behave. You do not rewrite the job description every time they answer a customer call. You let the standing instructions stay put and the customer-specific details show up in each conversation. Language models work the same way. The system message is the standing instructions; user messages are the per-call details. Rewriting the standing instructions every turn confuses the model and throws away a big cache discount the provider gives you for keeping prefixes stable.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
System-prompt stability is one of those production-grade prompting decisions that gets glossed over in tutorials but pays large dividends when you get it right. The system block and the user message look interchangeable in API documentation; in practice they serve different layers of the conversation contract and treating them as the same thing destroys both cost and behavior consistency.
This deep dive walks through why stable system prompts matter, what the provider cache discounts look like, how behavior consistency depends on stability, what belongs in the user message instead, and how to handle the rare case where a mid-conversation mode shift is the right move. The goal is a design framework an engineer can apply when wiring up a multi-turn conversational agent.
The punchline is two-axis. System is the standing orders that should not change. User is the per-turn details that should. Mixing the layers throws away the provider cache discount and produces inconsistent behavior; keeping them separate is the cheapest path to production-quality conversation.
Why provider caching makes stability cheap money
The major providers cache stable prompt prefixes. Anthropic's prompt caching is the clearest example: marked cacheable prefixes hit the cache on subsequent requests and are billed at roughly 10 percent of normal input cost, a 90 percent reduction. OpenAI's automatic prefix caching applies a 50 percent discount to matching prefixes without explicit markers. Gemini's context caching offers similar mechanics with per-token cache fees.
The cache hit requires the prefix to be byte-identical across requests. The system block is the prefix in most API call shapes, so keeping the system block stable across turns is exactly what triggers the cache hit. Varying the system block by even one token per turn invalidates the entire cached prefix on every turn and pays the full input cost on content that did not actually change.
On high-volume multi-turn routes the math is dramatic. A 5,000-token system block at full price across 100 turns costs 500,000 input tokens. Cached at 90 percent off across the same 100 turns, the same content costs 50,000 input token equivalents (the first turn pays full price; the rest pay 10 percent). The cost difference is roughly 10x on the system-block portion alone. Teams that have not designed for stability discover this on their first big bill.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Anthropic prompt caching offers about 90 percent off cached input tokens; production multi-turn routes lean on a stable system block to capture that discount.
- OpenAI automatic prefix caching applies 50 percent off matching prefixes; the design assumption is exactly that the system block does not change per turn.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow does Anthropic prompt caching cost structure shape multi-turn prompt design?
Cached input tokens are billed at roughly 10 percent of base; the design move is to push as much stable content as possible into the cacheable prefix and reserve user messages for the genuinely per-turn delta.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Editing the system prompt every turn with dynamic context, which invalidates the provider cache and produces inconsistent behavior across the conversation.
60 second bullets to scan on the way to the call.
Why stable system enables prompt caching
Provider-specific cache discounts and what they require
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.