When does inflating the system prompt actually lower cost per call?

Short answer·Medium·4.0 · 0·~3 min·Asked atAnthropic·Relevant atOpenAI Sglang

Attempt it

Counterintuitively, a much LONGER system prompt can drive down cost-per-call below a shorter one. Explain when this flips, and derive the rough hits-per-write threshold above which the verbose-but-cached prompt wins. Be concrete about the discount mechanism on at least one real provider (Anthropic or OpenAI).

Free · 2 AI evals / day

**Prompt caching** saves money by reusing the work the model did for the leading part of your prompt. The first time a long prompt arrives, the provider stores its internal state and charges a small premium for the write. Every subsequent call that starts with exactly the same bytes pays a deeply discounted read price for the cached part. This flips the usual rule that 'more tokens cost more'. A long, stable system prompt is more expensive on call one and cheaper on every call after that, because the cached part is billed at roughly ten percent of normal input price on Anthropic and around fifty percent on OpenAI. The break-even is the number of calls after which the long-cached prompt's total cost drops below the equivalent short-uncached version. The arithmetic is straightforward: total cost of N+1 calls with caching equals the write premium plus N times the read price; total cost of N+1 calls without caching equals (N+1) times the full input price of the short prompt. Solve for N and you get a small number, often single-digit calls, because the read discount on Anthropic is so large that a 5000-token prompt cached costs the same per hit as a 500-token uncached prompt. Prerequisites: the cached prefix must be byte-identical across calls (no per-user salt at the top), the call frequency must beat the cache TTL (5 minutes on Anthropic, various on OpenAI), and the cached content must be substantial enough to justify the write premium.

**The mechanism is prefix caching.** Providers store the KV-cache state for the leading bytes of a prompt; on a hit, the prefill work is skipped and the cached portion is billed at a deeply discounted rate. Anthropic charges roughly 10% of input price on a hit, with a ~25% premium on the initial write. OpenAI charges roughly 50% on a hit, with the write at normal input price (no premium in 2026). The asymmetry between write and read is what makes a long system prompt cheaper on average than a short one once the call count is high. Derive the break-even algebraically. Let L be the cacheable prefix length on the verbose option, S the length of the equivalent short prompt without caching, w the write multiplier (1.25 on Anthropic, 1.0 on OpenAI), r the read multiplier (0.1 on Anthropic, 0.5 on OpenAI), and N the number of cache hits between writes (i.e., the number of subsequent calls). Total cost of the cached option across N+1 calls is L*w + N*L*r input-equivalents. Total cost of the uncached option across the same N+1 calls is (N+1)*S input-equivalents. Set them equal and solve for N: N_break = (L*w - S) / (S - L*r) Apply Anthropic with L=5000, S=500: denominator = 500 - 500 = 0 and the formula diverges, but cumulatively the long option pays a write premium of `L*w - S = 5750` upfront, then matches the short option's per-call growth. It stays 5750 behind forever; the short prompt wins. On OpenAI with the same L and S: denominator = 500 - 2500 = -2000, which is the `S < L*r` regime where the long option grows faster per call and never breaks even. In cases where S is large relative to L*r, the break-even is finite and small. Example: L=2000, S=500 on Anthropic. Numerator = 2500 - 500 = 2000. Denominator = 500 - 200 = 300. N_break ≈ 7.

Quick reference

TL;DR

A long system prompt costs more on the first call (the cache write) and cheaper on every subsequent hit (the cache read at ~10% on Anthropic, ~50% on OpenAI).

Memory aid

Easy to grasp

Think of a gym with two memberships. The cheap one charges five dollars every time you walk in. The expensive one charges fifty dollars to sign up, then fifty cents per visit. If you visit once, the cheap one wins. If you visit a hundred times, the expensive one wins by a mile. Long cached prompts are the expensive membership: an upfront premium on the first call, then a tiny per-call charge after. Once you visit (call) enough times, it pays off, and that crossover usually comes after just a few calls because the per-hit discount is so steep.

Key concepts

Prompt caching Prefix cache Amortized cost

Detailed answer & concept explanation

~10 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Quick scan · 2–4 min

Everything important, quickly.

4 min: name prefix caching, identify write-vs-read pricing tiers, derive break-even N = (L*w - S)/(S - L*r), apply Anthropic and OpenAI numbers to a 5000-vs-500-token example, classify the workload by call rate vs TTL.

Real-world examples

Real products, models, and research that use this idea.

Anthropic's prompt-caching launch in 2024 reported up to 90% cost reduction on long-context workloads (RAG, coding agents) with break-even at 2-3 cache hits.
OpenAI's automatic prompt caching (2024+, GPT-4o and later) discounts ~50% on cached prefix tokens with no write premium, making the verbose-prompt win immediate from the second call.
Anthropic's extended-TTL tier (1 hour, 2026) targets slow-call-rate workloads where the default 5-minute TTL expires too often.
Anthropic's Computer Use SDK and Claude Code both rely on multi-kilo-token tool schemas in the system prompt; the verbose prompt is cheaper per agent step because the tool schemas cache across the entire session.
SGLang and vLLM self-hosted serving expose RadixAttention prefix caching with no write premium, so the verbose-prompt-wins regime is the steady-state behavior for any self-hosted long-prompt deployment.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QWhat happens to the break-even formula when output tokens are factored in?

QHow does Anthropic's extended-TTL tier change the workload classification?

QWhy is the boundary case (S = L*r) a loss zone for the verbose option on Anthropic with r=0.1?

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Treating prompt-caching value as a flat discount on every call. The actual structure has a write premium on the first call and a read discount on subsequent hits, and the break-even is amortized over the hit count, not per call.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

References

Primary sources. Skim if you want the original framing.