Prefill

Name the dominant cost driver of GPT-5.5 latency at a 200k-token input

Prefix-cache hit lands on a request: does speculative decoding still help that request?

TTFT: name the metric it captures and what dominates it

The roofline model in plain language, without the chart, what is it saying?

Prompt caching shows up on Anthropic and OpenAI pricing pages, what does it actually do?

Walk through what actually happens during the prefill phase of an LLM forward pass.

End-to-end latency: fill in the queue, prefill and decode components.

Why do hosted LLM APIs charge separate per-million rates for input and output tokens?

FLOPs vs FLOP/s: fill in the counts versus rate distinction.

Describe what happens during the decode phase of LLM inference.

Does Anthropic's message_start event count toward TTFT in production benchmarks?

Match TTFT and TPOT to the latency they measure during streaming

Prefill and decode: name the two phases of LLM inference and say which one is compute-bound

What problem does chunked prefill solve in production LLM serving?

How does prefix KV-cache sharing reduce serving cost across requests?

Which prompt-construction pattern is most likely to actually hit the provider's prompt cache?

Why does prefill saturate compute while decode is bottlenecked on memory bandwidth?

Which statement best explains why prefill and decode hit different bottlenecks on the same GPU?

Why does prefill's O(seq^2) attention cost saturate compute rather than bandwidth?

Why are output tokens typically 3-5× more expensive than input tokens?

Why does FlashAttention speed up serving even though it doesn't change the math?

Where does FlashAttention deliver the biggest serving speedup?