In a production LLM app with a stable system prompt + long retrieved context + variable user query, which part of the prompt becomes cacheable via Anthropic/OpenAI prompt caching? · Zenaique

Attempt it

Quick reference

TL;DR

The cacheable portion is the stable prefix from start of prompt up to the last byte-identical point across calls; the variable user query at the tail is not cached.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Imagine ordering at the same restaurant every day. The first time, the chef has to read the whole menu, prep the kitchen, and learn your preferences. Every day after, they remember the menu, the kitchen is ready, and they only need to hear what you want today. Prompt caching works the same way for an LLM. The system prompt and the long retrieved context are like the menu and the kitchen prep. They stay the same call after call, so the provider keeps them ready and you pay almost nothing for them on the next call. The new user query is the only fresh part the model has to process from scratch.

Key concepts

Prompt Caching Inference Optimization

Detailed answer & concept explanation

~5 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Sign in for 5 free unlocks/day or go Pro for unlimited. See Pro pricing.

Quick scan · 2–4 min

Everything important, quickly.

3 min: KV-cache reuse mechanism + cache boundary at last byte-identical point + Anthropic vs OpenAI pricing + byte-exact and TTL gotchas + how to measure hit rate.

Real-world examples

Real products, models, and research that use this idea.

Anthropic's prompt caching for Claude Opus 4.7 prices cached input at one tenth of normal cost, cutting agentic loop spend dramatically on stable system prompts plus tool definitions.
OpenAI's automatic prefix caching for GPT-5.5 fires on prefixes of 1024 tokens or more at roughly half price, with no explicit markers required.
Perplexity uses long-prefix caching on its retrieval and synthesis pipeline so repeated queries over a stable doc corpus pay only the suffix cost on follow-up turns.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QWhat's the right prompt ordering to maximize hit rate?

Sign in or go Pro to see the approach for this follow-up.

QHow would you measure cache hit rate in production?

Sign in or go Pro to see the approach for this follow-up.

QWhen does caching not help much?

Sign in or go Pro to see the approach for this follow-up.

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Assuming the cache fires on similar prompts when it requires a byte-identical prefix; even a session id injected at the top can knock the cache out.

Sign in or go Pro to see all red flags and common mistakes. See Pro pricing.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.