Prompt caching shows up on Anthropic and OpenAI pricing pages, what does it actually do?

When you send a prompt to an LLM, the server has to process the entire prompt before it can produce the first output token. That processing step is called **prefill** and it is expensive: the server runs the full forward pass over every prompt token and builds a KV cache to use during the rest of the request. **Prompt caching** is a feature that lets the server skip that prefill work for parts of the prompt that have been seen before. Concretely, the provider hashes the prefix of your prompt and stores the corresponding KV cache state server-side. The next time you send a request that starts with the same prefix, the provider recognizes the hash, loads the cached KV state, and only has to do fresh prefill on the new tokens at the end of the prompt. Two things change. **Cost drops** for the cached portion: Anthropic charges roughly 10 percent of the normal input price for cached reads (a 90 percent discount) and adds a 25 percent surcharge on the first write that creates the cache entry. OpenAI offers an automatic 50 percent discount on cached prefixes that are at least 1024 tokens long. **Latency drops** as well, because skipping prefill cuts time to first token by the same fraction that the cached portion represents. This matters most for applications with long, repeated context: chat applications with big system prompts, RAG systems that re-send the same retrieved chunks, agent frameworks that include tool descriptions in every call. For these patterns, prompt caching can cut both cost and TTFT by 70-90 percent.

**Prompt caching** is a provider-side feature where the LLM platform stores the KV cache state for a hashed prefix of your prompt and reuses it across requests that share the same prefix. When you send a new request, the provider checks whether its prefix matches a stored cache entry. If yes, it loads that KV state directly instead of recomputing it via prefill. The fresh prefill only runs on the tokens added after the cached prefix. Two concrete benefits, both on the input side. **Cost reduction**: Anthropic charges cached input tokens at roughly 10 percent of the normal input rate (a ~90% discount) and adds a ~25% surcharge on the cache-write that creates the entry. OpenAI offers a ~50% discount on cached input tokens, applied automatically when the same prefix of at least 1024 tokens appears on two requests within the cache TTL. **Latency reduction**: because the cached portion no longer goes through prefill, time to first token drops by roughly the same fraction. A request with a 10000-token system prompt followed by a 200-token user message could see TTFT fall from ~2000ms to ~200ms when the system prompt is cached. The ideal use cases are applications where a large portion of the prompt is reused across many requests: long system prompts in chat assistants, fixed instruction blocks in agent frameworks, the same retrieved chunks in a RAG pipeline, persistent conversation history. The cache key is the prefix as-tokenized, so any change near the start of the prompt invalidates the entire downstream cache. Order matters: put the stable, reused content at the front and the variable user input at the end. Output tokens are billed at the normal rate regardless of caching. Prompt caching is purely an input-side optimization.