Sequence length vs context window, what is the practical distinction and which one drives KV cache size?

Attempt it

Sequence length is the live token count of a given request, prompt plus output generated so far. Context window is the model's hard upper bound on sequence length, fixed by training and positional encoding (e.g., 200k for Claude Sonnet 4.6). KV cache memory scales linearly with the live sequence length, not with the context window, a 2k-token request on a 200k-context model only pays for 2k tokens of cache, not 200k.

Your answer

Quick reference

TL;DR

Context window is the model's fixed upper limit; sequence length is what the current request actually uses. KV cache scales with live sequence length, not the window.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Think of a parking garage that can hold 200 cars. That capacity is the context window: a fixed structural number that does not change. The sequence length is how many cars are parked right now. If only five cars are inside, you only pay for those five spots of upkeep, not for all 200. KV cache memory behaves the same way. The model architecture pre-declares the maximum it could ever store, but the GPU only allocates pages for the tokens you actually feed it. A small chat turn on a giant-context model is cheap. The total context window is only a ceiling, not a daily rent.

Key concepts

KV cache Context window Sequence length

Detailed answer & concept explanation

~6 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Sign in for 5 free unlocks/day or go Pro for unlimited. See Pro pricing.

Quick scan · 2–4 min

Everything important, quickly.

5 min: define both terms, show the KV cache memory formula, illustrate with a concrete 200k-context model serving a 500-token request, then close on why paged attention allocates on demand and what GQA and sliding window do to the slope in T.

Real-world examples

Real products, models, and research that use this idea.

Claude Opus 4.7 ships a 1M-token context window, but a typical chat turn is under 2k tokens, so most KV caches are tiny.
GPT-5.5 advertises 400k input tokens; pricing tiers reflect realized input length, not the ceiling.
Llama 4 Maverick supports 1M-context inference on H200 with FP8 KV cache, where serving cost scales with realized T.
vLLM and SGLang both allocate KV cache pages on demand at decode time, freeing them at sequence completion.
Gemini 3.1 Pro's 2M-context tier is priced per-token so long-document use cases pay only for tokens actually loaded.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QHow does grouped-query attention change the dependence of KV cache on sequence length?

Sign in or go Pro to see the approach for this follow-up.

QWhy does decode cost scale with full sequence length rather than just the new token?

Sign in or go Pro to see the approach for this follow-up.

QWhen does context window matter for cost even though sequence length is the runtime number?

Sign in or go Pro to see the approach for this follow-up.

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Assuming a 200k-context model always allocates 200k tokens of KV cache. The cache grows with the live request, not with the architectural cap.

Sign in or go Pro to see all red flags and common mistakes. See Pro pricing.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.