Sequence length vs context window, what is the practical distinction and which one drives KV cache size?
Context window is the model's fixed upper limit; sequence length is what the current request actually uses. KV cache scales with live sequence length, not the window.
Think of a parking garage that can hold 200 cars. That capacity is the context window: a fixed structural number that does not change. The sequence length is how many cars are parked right now. If only five cars are inside, you only pay for those five spots of upkeep, not for all 200. KV cache memory behaves the same way. The model architecture pre-declares the maximum it could ever store, but the GPU only allocates pages for the tokens you actually feed it. A small chat turn on a giant-context model is cheap. The total context window is only a ceiling, not a daily rent.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Context window and sequence length sound like synonyms but they answer different questions. Context window asks what is the model architecturally capable of holding. Sequence length asks what is the current request actually holding. Mixing them up leads to bad capacity planning and bad cost intuition.
This deep dive starts from the basic definitions, then derives the KV cache memory formula and shows which variable in it is the cost driver. From there we look at how production serving stacks actually allocate KV memory, why long-context models can advertise huge ceilings without making every request expensive, and how grouped-query attention and sliding-window attention bend the slope of cost against sequence length.
By the end you should be able to walk into a production review and explain why a 1M-context model is not 500 times more expensive to serve than a 2k-context model when the request distributions are similar.
Two definitions, drawn apart
Context window is a property of the model. It is set when the model is trained, baked into the positional encoding scheme (RoPE base, ALiBi slope set, learned-PE table size), and shipped as part of the model card. Claude Opus 4.7 ships 1M; GPT-5.5 ships 400k; Llama 4 Maverick ships 1M.
A model cannot serve a request longer than its context window without retraining or position-extension techniques like YaRN, NTK-aware interpolation, or Position Interpolation. The window is a hard architectural ceiling.
Sequence length is a property of a single in-flight request. It is the count of tokens currently held in that request's state, which is the prompt plus any output tokens generated so far. A user with a 100-token prompt asking for a 300-token reply will pass through sequence lengths from 100 up to 400 as decode proceeds.
The two numbers relate by one rule: sequence length is bounded above by context window. They are not the same number, and in production they differ by orders of magnitude.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Claude Opus 4.7 ships a 1M-token context window, but a typical chat turn is under 2k tokens, so most KV caches are tiny.
- GPT-5.5 advertises 400k input tokens; pricing tiers reflect realized input length, not the ceiling.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow does grouped-query attention change the dependence of KV cache on sequence length?
GQA reduces the number of distinct KV heads by a factor of g, so cache bytes scale as 2 * L * (H_q/g) * d_h * T * b. T still drives the slope, but the slope itself is smaller, which is why GQA models like Llama 3.1 70B can hold longer contexts on the same hardware.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Assuming a 200k-context model always allocates 200k tokens of KV cache. The cache grows with the live request, not with the architectural cap.
60 second bullets to scan on the way to the call.
Define context window as a model-level ceiling, not a runtime allocation
Define sequence length as the live per-request token count
Same topic, related formats. Practice these next.