Name the dominant cost driver of GPT-5.5 latency at a 200k-token input

Inference runs in two phases. Prefill processes the entire prompt to produce the first output token. Decode then emits the rest, one token at a time. Each phase has its own cost driver and each surfaces in a different latency metric. At 200k tokens of input, the prefill phase has a lot to do. Every prompt token has to flow through every layer of the model in one big parallel pass. Two pieces of work dominate. The MLPs do per-token work that scales linearly with the prompt length. The attention layers do work that grows with the prompt length, and in the naive implementation that attention scales as n times n, though efficient implementations like FlashAttention bring this closer to linear-in-n while still moving a lot of data. The net effect is that prefill at 200k tokens takes multiple seconds, even on top-of-line hardware. That wait is what the user sees as TTFT, time to first token. Prefill also leaves behind a KV cache, the keys and values for every one of those 200k positions, so the model does not recompute them during decode. On a 70B-class model, that cache can be tens of gigabytes. Then decode begins. Each output token requires one full forward pass through the model. That pass has to read the full model weights from HBM and the full KV cache from HBM. The weights are constant per step; the KV cache is now huge because of the long prompt. Per-step time is essentially total bytes loaded divided by HBM bandwidth. At 2k context the KV is negligible and decode runs at the model's normal speed. At 200k context the KV is many gigabytes and decode visibly slows. The output side, 500 tokens generated, costs almost nothing in comparison. Even at slower per-token decode, 500 tokens at maybe 50 ms each is 25 seconds of wall-clock and is billed at the output rate. The prefill plus KV bandwidth dominate both the time and the cost on this profile.

Two distinct cost mechanisms, two distinct latency metrics. The 200k prompt sets the cost of prefill (which surfaces as TTFT) and the cost of decode (because the KV cache it produces makes every later step slower). The 500 output tokens are a third axis that barely shows up on this profile. Prefill is one parallel forward pass over the prompt. The attention layer in prefill operates as a matrix-matrix multiplication across all 200k positions, with arithmetic that scales like n^2 in the naive implementation; FlashAttention brings the memory traffic down to linear-in-n while keeping the compute at n^2 done efficiently. The MLP layers do per-token work linear in n. The net is that prefill compute grows super-linearly with prompt length and on a frontier model at 200k input it takes multiple seconds even on H200-class GPUs. That entire wait lands in TTFT: the user sees nothing until prefill finishes and the first decode step produces the first token. Prefill also writes the KV cache as a side effect. For every layer, every position, every KV head, it stores K and V vectors. For a 70B model with GQA-8 (80 layers, 8 KV heads, 128 head dim, FP16): per-token KV ≈ 80 * 8 * 128 * 2 * 2 = 320 KB per token. At 200k tokens that is about 64 GB of KV cache sitting in HBM after prefill. Some of that gets sharded across GPUs in tensor-parallel deployments, but the aggregate is the same. Decode then becomes bandwidth-bound by both the weights and the KV. Each step loads the full model weights (constant, around 140 GB at FP16 for 70B) plus the full live KV cache (now 64 GB) from HBM. At 3 TB/s HBM bandwidth: weight contribution ≈ 47 ms, KV contribution ≈ 21 ms, total ≈ 68 ms per token. At 2k context the KV is 0.65 GB, negligible, and per-token sits at the 47 ms baseline. So per-step decode at 200k is roughly 45 percent slower than at 2k on the same model and hardware.