Short Answer

What problem does PagedAttention solve and why does block-based allocation enable larger batches?

How do online (interactive) and offline (batch) inference workloads differ in optimization target?

What traces and metrics do you need to debug a P99 TTFT regression in production LLM serving?

Why is the vision-encoder phase a different bottleneck class from LLM decode in a multi-modal model?

How does S-LoRA serve thousands of LoRA adapters concurrently on a single base model?

How does DeepSeek's Multi-Latent Attention (MLA) compress the KV cache below GQA?

Why is HBM bandwidth the binding constraint for LLM decode and how do you reason about the 'memory wall'?

How are RoPE NTK scaling and YaRN used at inference time to extend context beyond the training length?

Why does naive sliding-window KV eviction break and how does StreamingLLM's attention-sink fix it?

How much does FP8 KV cache help, what does it cost and what is KIVI?

What exactly does the KV cache store and what computational redundancy does it eliminate?

Compute the KV cache size for a 70B model at 128k context (80 layers, 64 heads, head_dim 128, FP16).

Why are output tokens typically 3-5× more expensive than input tokens?

For a 64-Q-head model with GQA group size G=8, how many K/V heads exist and what is the cache reduction vs MHA?

Why is 'reduce FLOPs to speed up decode' the most common beginner misconception in LLM serving?

Why does FlashAttention speed up serving even though it doesn't change the math?

What is GGUF and how do llama.cpp / mlc-llm enable on-device inference where vLLM cannot run?

What problem do DistServe and Splitwise solve by separating prefill and decode onto different GPUs?

Compare greedy, sampling and beam search at inference: cost and best use case.

For a 200-token prompt and 1000-token completion, why does decode dominate end-to-end latency?

Walk through the byte accounting that proves a single-batch decode step is bandwidth-bound on H100.

Why did continuous batching replace static batching for LLM serving?

What does the vLLM-style continuous-batching scheduler actually do at each step?

How does Sarathi-Serve's chunked prefill change the latency profile of a serving system?

Why is batching the single biggest throughput lever for LLM decode?