Name the dominant cost driver of GPT-5.5 latency at a 200k-token input

Short answer·Medium·4.0 · 0·~3 min·Asked atOpenAI

Attempt it

A team is benchmarking GPT-5.5 at 200,000 input tokens and 500 output tokens. End-to-end latency is dominated not by the 500 output tokens but by the prompt itself, with TTFT in the multi-second range and decode steps that get measurably slower than at a 2k-token context. Identify the two distinct cost drivers and explain why one shows up in TTFT and the other in per-token decode time.

Free · 2 AI evals / day

Inference runs in two phases. Prefill processes the entire prompt to produce the first output token. Decode then emits the rest, one token at a time. Each phase has its own cost driver and each surfaces in a different latency metric. At 200k tokens of input, the prefill phase has a lot to do. Every prompt token has to flow through every layer of the model in one big parallel pass. Two pieces of work dominate. The MLPs do per-token work that scales linearly with the prompt length. The attention layers do work that grows with the prompt length, and in the naive implementation that attention scales as n times n, though efficient implementations like FlashAttention bring this closer to linear-in-n while still moving a lot of data. The net effect is that prefill at 200k tokens takes multiple seconds, even on top-of-line hardware. That wait is what the user sees as TTFT, time to first token. Prefill also leaves behind a KV cache, the keys and values for every one of those 200k positions, so the model does not recompute them during decode. On a 70B-class model, that cache can be tens of gigabytes. Then decode begins. Each output token requires one full forward pass through the model. That pass has to read the full model weights from HBM and the full KV cache from HBM. The weights are constant per step; the KV cache is now huge because of the long prompt. Per-step time is essentially total bytes loaded divided by HBM bandwidth. At 2k context the KV is negligible and decode runs at the model's normal speed. At 200k context the KV is many gigabytes and decode visibly slows. The output side, 500 tokens generated, costs almost nothing in comparison. Even at slower per-token decode, 500 tokens at maybe 50 ms each is 25 seconds of wall-clock and is billed at the output rate. The prefill plus KV bandwidth dominate both the time and the cost on this profile.

Two distinct cost mechanisms, two distinct latency metrics. The 200k prompt sets the cost of prefill (which surfaces as TTFT) and the cost of decode (because the KV cache it produces makes every later step slower). The 500 output tokens are a third axis that barely shows up on this profile. Prefill is one parallel forward pass over the prompt. The attention layer in prefill operates as a matrix-matrix multiplication across all 200k positions, with arithmetic that scales like n^2 in the naive implementation; FlashAttention brings the memory traffic down to linear-in-n while keeping the compute at n^2 done efficiently. The MLP layers do per-token work linear in n. The net is that prefill compute grows super-linearly with prompt length and on a frontier model at 200k input it takes multiple seconds even on H200-class GPUs. That entire wait lands in TTFT: the user sees nothing until prefill finishes and the first decode step produces the first token. Prefill also writes the KV cache as a side effect. For every layer, every position, every KV head, it stores K and V vectors. For a 70B model with GQA-8 (80 layers, 8 KV heads, 128 head dim, FP16): per-token KV ≈ 80 * 8 * 128 * 2 * 2 = 320 KB per token. At 200k tokens that is about 64 GB of KV cache sitting in HBM after prefill. Some of that gets sharded across GPUs in tensor-parallel deployments, but the aggregate is the same. Decode then becomes bandwidth-bound by both the weights and the KV. Each step loads the full model weights (constant, around 140 GB at FP16 for 70B) plus the full live KV cache (now 64 GB) from HBM. At 3 TB/s HBM bandwidth: weight contribution ≈ 47 ms, KV contribution ≈ 21 ms, total ≈ 68 ms per token. At 2k context the KV is 0.65 GB, negligible, and per-token sits at the 47 ms baseline. So per-step decode at 200k is roughly 45 percent slower than at 2k on the same model and hardware.

Quick reference

TL;DR

At 200k input the bill is paid by prefill compute (which sets TTFT) and by the resulting KV cache size (which slows every decode step). Output tokens are a rounding error on this profile.

Memory aid

Easy to grasp

Imagine you walk into a library and hand the librarian a 200-page document and ask for a one-paragraph summary. Before the librarian can write a single word, they have to read all 200 pages. That reading time is what you wait for at the very beginning, even though no answer has appeared yet. Now while they write the paragraph, every sentence has to be checked against what they remember from the 200 pages, which is a huge stack of notes. So each sentence comes out a bit slower than if the document had been 2 pages. The summary itself is short and cheap; what is expensive is the original reading and the constant flipping through notes.

Key concepts

Prefill phase Decode phase KV cache Time to first token Long-context inference

Detailed answer & concept explanation

~7 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Quick scan · 2–4 min

Everything important, quickly.

4 min: name both phases (prefill drives TTFT, KV-amplified decode drives TPOT), quantify both for a 70B model at 200k, explain why output cost is small here, and rank the mitigation stack (prompt caching first, then KV quant).

Side-by-side

Latency component	Dominant driver at 200k	What surfaces it	Top mitigation
TTFT (prefill)	Per-token MLP + attention compute over 200k positions	Multi-second wait before first token	Prompt caching of stable prefix
TPOT (decode)	Per-step weight + KV bytes loaded from HBM	Each decode step ~45% slower than at 2k	FP8 KV cache, GQA/MLA, TP
Output token cost	Small share of bill at 200k:500 ratio	Routine decode latency for 500 tokens	Speculative decoding (weak lever here)
Dollar bill	Input tokens (volume) + KV-amplified decode time	Input share dominates total spend	Prompt caching (cuts input cost by ~90%)

Real-world examples

Real products, models, and research that use this idea.

Anthropic Claude Opus 4.7 at 200k context shows TTFT in the 5-15 second range and per-token decode about 40-60 percent slower than at 4k; the same prompt with cache hit returns the first token in under a second.
GPT-5.5 long-context benchmarks at 200k routinely report 80-95 percent of cost going to prefill input tokens, even though output is billed at higher per-token rate, simply because there are 400x more input tokens.
NVIDIA's TensorRT-LLM ships FP8 KV cache as a switch; teams that flip it on for long-context workloads see decode TPOT drop 30-50 percent at 100k+ contexts with imperceptible quality impact.
Mooncake disaggregates prefill and decode pools for Kimi's long-context serving, with the prefill pool sized for compute throughput and the decode pool sized for KV-cache HBM capacity, the two profiles being so different they justify separate hardware tiers.
DeepSeek V4 with Multi-head Latent Attention compresses KV to a low-rank latent, dropping the per-token KV bytes by roughly 90 percent versus standard MHA, which is what makes its 128k context serve at competitive per-token decode latency.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QHow does FlashAttention change the prefill cost at 200k?

QIf you had to pick one optimization for this exact workload, what would it be and why?

QWhy does the KV cache term sometimes exceed the weights term at very long context?

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Thinking the output token count drives long-context latency. At 200k input and 500 output, the prompt processing and the KV-amplified decode dominate; the 500 output tokens contribute single-digit percent of wall-clock and bill.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

References

Primary sources. Skim if you want the original framing.