Record cache_read_input_tokens and cache_creation_input_tokens as separate span attributes alongside fresh input_tokens; the cost calculator applies the 10 percent cache-hit price to cache reads and dashboards expose
Imagine a coffee shop where the first time you order a complicated drink, the barista has to make the syrup from scratch (expensive), but after that they keep a jar of your syrup behind the counter and just stir it into milk (cheap). The cash register has to tell you which one happened, otherwise you keep paying the expensive price even when they used the cheap jar. LLM prompt caching is the same: the first call writes a cache, follow-ups read from it at one-tenth the price. The trace on every call has to show three numbers separately, not one combined number: how many tokens were fresh (full price), how many were read from cache (cheap), and how many created a cache for later. Without those three numbers, your cost dashboard cannot tell you whether the cache is actually saving money.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Prompt caching, introduced by Anthropic in mid-2024 and adopted by OpenAI and Google shortly after, is one of the highest-leverage cost optimizations in modern LLM systems. Cached tokens cost a small fraction of fresh input tokens (typically 10 percent on Anthropic, roughly 50 percent on OpenAI), and on workloads with a long stable prefix (chat with system prompt, few-shot examples, retrieved context) the cache hit rate can exceed 90 percent.
The observability side of this is unglamorous but load-bearing. If the spans do not break input tokens into three buckets (fresh, cache read, cache creation), the cost calculator silently over-attributes by a large multiple and the team cannot prove that the caching work is paying off.
The three-bucket cost model
Every modern LLM provider that supports prompt caching returns usage information in three input buckets plus the standard output bucket:
- Fresh input tokens. Tokens not covered by any cached prefix. Billed at the model's normal input price.
- Cache read tokens. Tokens in the request that hit a cached prefix. Billed at a deep discount (10 percent on Anthropic in 2026, around 50 percent on OpenAI's gpt-5.5 family).
- Cache creation tokens. Tokens that the provider had to write into the cache because they were not previously cached. Anthropic charges a 25 percent premium (1.25x the input price) for cache creation; OpenAI charges the normal input price.
Output tokens are unchanged.
Why the price differs per bucket
Cache reads are cheap because the provider can short-circuit a large fraction of the prefill compute. Cache creation is expensive (or premium-priced) because the provider has to do the prefill and store the KV cache for future reuse. Fresh input is the baseline. The structure of the price model dictates the structure of the attribution.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
| Token bucket | Anthropic 2026 price ratio | OpenAI 2026 price ratio | Span attribute |
|---|---|---|---|
| Fresh input | 1.00x | 1.00x | gen_ai.usage.input_tokens (total, includes cache) |
| Cache read | 0.10x | approximately 0.50x | gen_ai.usage.cache_read_input_tokens |
| Cache creation | 1.25x | 1.00x | gen_ai.usage.cache_creation_input_tokens |
| Output | 1.00x (output price) | 1.00x (output price) | gen_ai.usage.output_tokens |
Real products, models, and research that use this idea.
- Anthropic's prompt-caching feature returns usage.cache_read_input_tokens and usage.cache_creation_input_tokens alongside usage.input_tokens; the OTel gen_ai semconv mirrors these names.
- OpenAI's prompt caching (gpt-5.5 family and later) returns usage.prompt_tokens_details.cached_tokens; the cost ratio is roughly 50 percent of fresh input.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow do you reconcile span-level token counts with the provider's billing invoice end of month?
Sum each bucket across all traces in the billing period, multiply by current prices, compare to the provider invoice. Build a CI test that runs the reconciliation on a sample day and alerts on more than 2 percent drift; drift usually means a provider pricing change.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Collapsing cache_read_input_tokens into a single input_tokens field so the cost calculator over-attributes by 10x and the cache_hit_rate metric is invisible.
60 second bullets to scan on the way to the call.
The three input-token buckets (fresh, cache read, cache creation) and their typical price ratios
OTel gen_ai semconv attribute names for each bucket
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.