KV Cache

Which prompt-construction pattern is most likely to actually hit the provider's prompt cache?

What problem does PagedAttention solve and why does block-based allocation enable larger batches?

What is the primary mechanism by which PagedAttention increases serving throughput?

What traces and metrics do you need to debug a P99 TTFT regression in production LLM serving?

Which metrics are essential to debug a P99 TTFT regression in production LLM serving?

How does DeepSeek's Multi-Latent Attention (MLA) compress the KV cache below GQA?

Match MLA, GQA and MQA to their mechanisms, cache reduction and quality posture

What exactly does the KV cache store and what computational redundancy does it eliminate?

Compute the KV cache size for a 70B model at 128k context (80 layers, 64 heads, head_dim 128, FP16).

Predict the KV cache size in GB for given model dimensions

Which statement most accurately captures why GQA, not MQA, became the production default?

Match each attention variant to its KV-cache property and a production user

For a 64-Q-head model with GQA group size G=8, how many K/V heads exist and what is the cache reduction vs MHA?

What problem do DistServe and Splitwise solve by separating prefill and decode onto different GPUs?

Compare greedy, sampling and beam search at inference: cost and best use case.

Why do production chat APIs almost never offer beam search as a decoding option?

Why did continuous batching replace static batching for LLM serving?

What does the vLLM-style continuous-batching scheduler actually do at each step?

Put the vLLM-style continuous-batching scheduler steps in correct order for one iteration

Walk through paged attention end to end, page table, block lookup, and how it enables higher throughput.

What problem does paged attention (vLLM) solve, and what OS concept does it borrow from?

Compare MHA, MQA, GQA, MLA, what production tradeoff are they all addressing, and which models use each?

Match each attention variant to its KV-cache memory reduction strategy.

Why does autoregressive generation use a KV cache?

What does the KV cache store, and why does it avoid Q?