A reasoning request prefills 512 prompt tokens, then autoregressively generates 4,000 thinking tokens followed by 300 answer tokens. KV cache stores one key/value pair per generated token (prefill included in the cache before decode). How many tokens total sit in the KV cache after generation completes?
KV cache holds 512 prefill + 4,000 thinking + 300 answer tokens = 4,812 total after generation completes.
Think of the KV cache like a growing notebook where the model writes down every word it has read or generated so it does not re-read from scratch. The prompt pages go in first, then every thinking word, then every answer word. Count all pages in the notebook — not just the thinking chapter.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Long chain-of-thought breaks serving economics through KV cache growth. This predict-output question tests whether you count every token in the sequence — not just the visible answer.
We walk prefill, decode, and the arithmetic behind 4,812.
What the KV cache stores
During autoregressive generation, transformers cache key and value projections for prior tokens so each new token attends to history without recomputing past K/V. Cache length T grows by one per processed token.
Prefill phase processes the prompt in parallel (often batched attention), populating cache for all prompt positions. Decode phase generates one token at a time, appending to cache each step.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
# KV cache token count after full generation
prefill = 512
thinking = 4000
answer = 300
total_kv_tokens = prefill + thinking + answer # 4812Real products, models, and research that use this idea.
- OpenAI o-series thinking tokens bill separately because they extend decode length and cache footprint.
- vLLM and TensorRT-LLM paging evict KV blocks when long CoT blows per-request memory.
What an interviewer would ask next. Try answering before peeking at the approach.
QEstimate HBM for T=4812 on a 70B GQA model with concrete dims.
Plug L, H_kv, d_h, b into the bytes formula and convert to GB.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Counting only thinking tokens (4,000) or only generated tokens (4,300) while omitting the 512-token prefill.
60 second bullets to scan on the way to the call.
Prefill tokens enter cache before decode
Each decode step appends one token
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.