Walk through the prompt structure that maximizes provider prompt cache hit rate

Flashcard·Medium·4.0 · 0·~30s·Asked atCloudflare Midjourney Two Sigma

Attempt it

Order matters because the cache is prefix-based. Put the most stable content first, the least stable content last.

For a typical RAG chat: system prompt (rarely changes, cache it), then a stable tool or format spec (cache it), then few-shot examples if used (cache them), then retrieved context (cache only if the same documents recur, e.g., per-tenant knowledge bases), then conversation history (variable), then the latest user message (never cached).

Mark cache_control on the last stable boundary; everything before it gets cached and re-used at the ~90 percent discount. Avoid interpolating dynamic content (timestamps, user ids) into the cached prefix; one different byte in the prefix invalidates the cache. Per-tenant cache keys come naturally if the system prompt or stable context contains the tenant id at a marked break.

Your answer

Provider **prompt caching** lets you re-use a long stable prefix at roughly 10 percent of normal input-token cost. The catch is that caching is **prefix-based**, not content-based. The cache key is the exact byte sequence from the start of the prompt up to a marked boundary; one different byte invalidates the entire cached prefix. The design rule that follows: **sort content by stability descending**. Most stable on top, least stable at the bottom. For a typical RAG chat application, that ordering is: 1. **System prompt**, rarely changes. Cache it. 2. **Tool or format spec**, stable across requests. Cache it. 3. **Few-shot examples**, stable for the prompt version. Cache them. 4. **Retrieved context**, cache only if the same documents recur across requests (per-tenant knowledge bases, frequently-referenced docs). 5. **Conversation history**, variable per session. 6. **Latest user message**, never cached. The `cache_control` marker goes on the last stable boundary. Everything before the marker is cached and re-used at the discounted rate; everything after is treated as fresh input. The most common mistake is interpolating dynamic content into the cached section, a current timestamp in the system prompt, a request id in the tool spec, a user name in the few-shot section. Each one breaks the prefix on every request and the cache never hits.

Caching is prefix-based, so the design is dominated by **content ordering and clean stability boundaries**. The canonical RAG-chat layout: 1. System prompt (changes rarely, per prompt version), cached. 2. Tool definitions or output-format spec, cached. 3. Few-shot examples, cached. 4. Retrieved context, cached only when documents recur (per-tenant KB) or when explicitly designed for reuse. 5. Conversation history, typically variable per session; cache only if conversation prefix stability is worth the complexity. 6. Latest user message, never cached. Mark `cache_control` (Anthropic) or use the equivalent provider mechanism on the last stable boundary. Anthropic in 2026 allows up to 4 cache breakpoints, which enables nested layers (cache the system, then cache system+tools, then cache system+tools+context). OpenAI prompt caching is automatic on prefixes ≥1024 tokens with no explicit marker. The key discipline is **byte-exact prefix stability**. Any of these silently breaks the cache: interpolating a timestamp, a request id, a per-request user id, a freshly-generated UUID, current date in the system prompt, varying whitespace, varying JSON key order in serialized tool definitions. Audit the cached prefix as a fixed byte string and treat any change to it as a deploy event. Per-tenant caching emerges naturally if the tenant id sits at a marked break: cache the global system prompt, then cache global system plus tenant-specific context (each tenant gets its own cache entry for that level). Economics: a typical RAG application with a 3000-token cached prefix and a 200-token variable suffix sees cached input billed at ~10 percent of normal; total input cost drops by 70 to 85 percent depending on the prefix to suffix ratio.

TL;DR

Sort the prompt by stability descending, stable system and tools first, retrieval and history next, user input last, and mark cache_control on the last stable boundary.

Memory aid

Easy to grasp

Imagine writing a letter where the top half is a form everyone signs and the bottom half is personal. If the form part is identical every time, the print shop can keep a stack pre-printed and only handwrite the bottom for each customer. That is provider prompt caching. The trick is to put the unchanging form on top and the personal note at the bottom. If a single date or name leaks into the form section, the pre-printed stack is useless because every letter looks different at the top. Sort by what changes least to what changes most, and put the cache marker just before the personal part starts.

Key concepts

LLMOps Context Engineering

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Provider prompt caching is one of the largest cost-reduction levers available to LLM applications in 2026. Anthropic's, OpenAI's, and Google's implementations differ in the API surface, but they share the same core mechanic: a stable prefix billed at a heavy discount on repeated reads.

The entire engineering exercise is structuring the prompt so the stable prefix is large and the variable suffix is small, and then defending the prefix's byte-exact stability against the dozens of ways application code can accidentally break it.

Why prefix-based caching shapes the design

The cache key is the byte sequence from position zero up to a marked boundary in the prompt. Either you mark the boundary explicitly (Anthropic's cache_control) or the provider chooses it automatically based on a minimum-length threshold (OpenAI's 1024-token rule).

Either way, the invariant is the same: one different byte before the boundary invalidates the entire prefix. This is not content-similarity caching; it is exact-prefix matching.

The design implication is single-axis: sort content from most stable to least stable, top to bottom. Anything that varies per request must come after the cache boundary. Anything that stays fixed across many requests should come before.

The canonical layout for a RAG chat application

The prefix-stability defense

Per-tenant caching and cache cardinality

Write vs read economics and break-even

Provider differences in 2026

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Real products, models, and research that use this idea.

Anthropic prompt caching ships with explicit cache_control markers and up to 4 breakpoints, with 5-minute and 1-hour TTL tiers in 2026.
OpenAI prompt caching applies automatically to prefixes of 1024 tokens or more, billed at ~50 percent of the uncached rate.

What an interviewer would ask next. Try answering before peeking at the approach.

QHow would you measure the dollar impact of enabling caching on a 100M-request per month workload?

Compute baseline input-token cost, measure cached-input ratio from provider response telemetry post-rollout, multiply the rate delta against cached tokens; expect 50-80 percent input cost reduction.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Interpolating a timestamp or request id into the system prompt and silently invalidating the cache on every single request.

60 second bullets to scan on the way to the call.

Why caching is prefix-based and what that implies for content ordering
The canonical RAG-chat layout from most to least stable

Primary sources. Browse if you want the original framing.