Pick the metric that best signals you are getting value from provider prompt caching

MCQ·Medium·4.0 · 0·~1 min·Asked atAirbnb Crewai Tesla

Attempt it

**Provider prompt caching** (Anthropic, OpenAI, Google) lets you mark a long stable prefix of your prompt (system prompt, few-shot examples, tool definitions) as cacheable. The first request pays full price to write the prefix into the cache. Subsequent requests that share the same prefix are billed at a steep discount, often around 10 percent of normal input pricing. **The metric that directly measures whether caching is working** is the **cache-hit rate on cached input tokens**: cached prefix bytes divided by total prefix bytes served, usually expressed as a percentage. The other options: - **Total monthly token spend** moves for many reasons (traffic growth, prompt size changes, model mix). A drop could be caching, but it could also be lower traffic. Not isolating. - **Unique users per day** is orthogonal; it has nothing to do with whether caching is firing. - **P99 latency to first byte** does improve when cache hits because the model skips re-processing the cached prefix, but it is a side effect, not the direct value signal. Latency moves with many other factors (provider load, model choice, context length). The practical dashboard pairs **hit rate** with **a list of the prompts that are NOT hitting cache**. Those are the prompts where the prefix changes every request (random user info early in the prompt, timestamps, request-specific context inserted before the stable part). Rewriting them to push variable content to the end and stable content to the front is where prompt-cache work pays off.

**The right answer is cache-hit rate on cached input tokens** because that is the only metric on the list that isolates cache contribution. **Why cache-hit rate works.** Anthropic and OpenAI both bill cached input tokens at roughly 10 percent of normal input pricing. The relationship between hit rate and savings is direct: each percentage point of hit rate is a corresponding cut on input-token cost for that fraction of traffic. The math is $$\text{effective input cost} = \text{normal input price} \cdot (1 - \text{hit rate}) + \text{normal input price} \cdot 0.1 \cdot \text{hit rate}$$ A hit rate of 80 percent on a workload where input tokens dominate cost (typical for long system prompt agents) saves roughly 72 percent on input cost. Hit rate is the lever. **Why the others are noisy or orthogonal.** - **Total monthly token spend** moves with traffic, prompt size, model mix, output length, and several other variables. Spend trending down could mean caching is working, or it could mean traffic dropped, or it could mean a routing change moved load to a cheaper model. Spend alone does not isolate the cache contribution. - **Unique users per day** is a product metric, not a caching metric. The two have no causal relationship. - **P99 TTFT** does improve with cache hits because the model can skip re-processing the cached prefix, but TTFT also depends on provider load, model selection, output length, queueing, and network. It is a downstream side effect, not a direct cache-effectiveness measure. **Companion metric.** The hit-rate dashboard should also show **what is not getting cached**: prompts whose prefix changes every request (timestamps inserted early, per-user content before the stable block, slight format variations that break the cache key). Those are the rewrite candidates that lift the headline hit rate.

TL;DR

Cache-hit rate on cached input tokens directly measures value because cached tokens are billed at roughly 10 percent of normal. Total spend, user count, and TTFT are downstream or noisy proxies.

Memory aid

Easy to grasp

Imagine you get a 90 percent discount on any grocery you bought before. The right question to ask is 'what fraction of my groceries qualified for the discount this week?' That ratio tells you directly how much you saved. If you only look at your total bill, it goes up and down for many reasons (you bought more, prices changed, you tried new items) and you cannot tell whether the discount program is doing its job. Cached prompt tokens work the same way. The right dashboard shows hit rate. Bonus: the same dashboard should also show what is not getting cached, so you can rewrite those prompts to share more prefix and lift the rate.

Key concepts

LLMOps

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Provider prompt caching is one of the largest LLM cost levers available in 2026, often cutting input-token spend by half or more on workloads with long stable system prompts. But the lever is only useful if you can measure whether it is working, and the right measurement is less obvious than it looks. Teams new to the feature often watch total spend or TTFT and conclude one of two wrong things: either caching is not helping (because spend moved up with traffic) or caching is fully working (because TTFT improved for unrelated reasons).

This walkthrough explains why cache-hit rate on cached input tokens is the right value signal, the economics that make it linear with savings, why the alternative metrics on the list are confounded, the companion 'what is not getting cached' view that drives the action loop, and the prefix-stability discipline that decides whether caching fires at all.

Why cache-hit rate is the direct value signal

Cache-hit rate on cached input tokens is defined as cached prefix bytes served divided by total prefix bytes served, expressed as a percentage. Providers expose this in the API response (Anthropic's usage.cache_read_input_tokens and usage.cache_creation_input_tokens; OpenAI's usage.prompt_tokens_details.cached_tokens). You aggregate per route or per template to get the rate.

The relationship to savings is linear and direct. Cached input tokens bill at roughly 10 percent of normal input pricing on both Anthropic and OpenAI in 2026. The effective input cost is

C_{\text{eff}} = C_{\text{normal}} \cdot (1 - h) + C_{\text{normal}} \cdot 0.1 \cdot h

where h is the hit rate. Simplifying, C_eff = C_normal * (1 - 0.9*h). At h = 0.8, effective input cost is 28 percent of normal. At h = 0.9, 19 percent. At h = 0.5, 55 percent.

This is the only metric on the list with that property. Moving hit rate from 50 to 80 percent on a workload where input dominates cost is a real, measurable, attributable savings event. You can pre-compute the expected savings before shipping a prompt rewrite, ship it, and verify hit rate moved as predicted.

Cache-write cost. Writing to the cache costs slightly more than normal input tokens on first use (Anthropic charges about 1.25x normal). The first request that warms a prefix pays this premium; subsequent requests within the cache TTL (5 minutes on Anthropic) get the discount. Hit rate captures the post-warm steady state; the cache-write premium is amortized across the warm period and is small in workloads with steady traffic on a given route.

Why the alternatives are confounded or orthogonal

The companion view: what is not getting cached

Prefix-stability discipline and the failure mode

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Real products, models, and research that use this idea.

Anthropic's prompt-caching feature for Claude charges cached input tokens at roughly 10 percent of normal input price, with cache-hit signal in the API response.
OpenAI's automatic prompt caching for GPT-5.5 and beyond similarly discounts cached prefix tokens and exposes hit signal in the response usage object.

What an interviewer would ask next. Try answering before peeking at the approach.

QHow would you design a prompt template to maximize cache hit rate for a multi-tenant agent?

Keep the first several thousand tokens identical across all tenants and all requests: shared system prompt, shared tool catalog, shared few-shot examples. Put tenant-specific config and user-specific content in the user message or in a tail section after the stable block. Document the prefix-stability contract so refactors do not silently break it. Measure per-route hit rate to verify.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Watching total monthly spend as the cache-value signal. Spend moves with traffic and prompt size; only hit rate isolates the caching contribution.

60 second bullets to scan on the way to the call.

Why cache-hit rate on cached input tokens is the direct value signal
What cached input tokens are billed at relative to normal

Primary sources. Browse if you want the original framing.