🎯Today's Daily Challenge, completing this counts toward your streak.Today's set →

vLLM defaults to a page size of 16. Pick what makes 1 and 1024 worse choices.

MCQ·Medium·4.0 · 0·~1 min·Relevant atNVIDIA

Attempt it

**PagedAttention** is the trick vLLM uses to manage the KV cache the way operating systems manage virtual memory. Instead of allocating one big contiguous chunk per request (the old way, which wastes a lot of memory when requests have different lengths), vLLM divides the GPU's KV memory into fixed-size **blocks** and assigns blocks to requests on demand. A per-request **block table** maps logical positions to physical blocks. The block size B is the parameter the question is asking about. vLLM defaults to B = 16. **Why B = 1 is bad.** Every single token attended to would require a separate block-table lookup. The attention kernel would do a memory-pointer chase per token, which is the worst possible pattern for GPU memory. GPUs reach peak bandwidth by reading contiguous chunks of memory in coalesced fashion; reading one token at a time defeats that completely. Latency explodes. **Why B = 1024 is bad.** A request that generates only 30 tokens still gets allocated one full 1024-token block. The remaining 994 slots in the block are wasted. This is **internal fragmentation**: memory you allocated but cannot use. With realistic generation-length distributions, B = 1024 wastes most of the KV cache, which is the exact thing PagedAttention exists to avoid. **Why B = 16 hits the sweet spot.** Sixteen tokens is enough for the GPU to do one coalesced read per block (no per-token overhead) while keeping the wasted-slot count per request under 16. For typical generation lengths (50-500 tokens), the fragmentation is under 10%. The choice between 8, 16, and 32 is more debatable; the choice between 1 and 16, or between 16 and 1024, is not.

**PagedAttention** is the OS-style memory management vLLM applies to the KV cache. The cache is partitioned into physical blocks of size B (tokens), and each request gets a logical block table mapping its positions to physical blocks on demand. The choice of B is a classic capacity vs overhead curve, and both extremes are bad for distinct reasons. **Why B = 1 fails.** Every position attended to costs a separate block-table lookup. The attention kernel becomes a sequence of indirect memory accesses, one per token, completely defeating GPU coalesced reads. Tensor-core matmul kernels expect contiguous K and V blocks; with B = 1, every read is a separate small fetch. Bandwidth utilization drops by an order of magnitude. **Why B = 1024 fails.** Internal fragmentation. A request that generates 30 tokens gets one block of 1024 slots, wasting 994. Across a fleet handling thousands of concurrent requests with diverse generation lengths, this wastes most of the KV cache. The entire motivation for PagedAttention is to reduce the wasted memory from naive contiguous-allocation schemes; B = 1024 reintroduces the same problem at coarser granularity. **Why B = 16 wins.** A 16-token block is enough for the GPU to do one coalesced load per block and amortize the page-table lookup over 16 attentions. Empirically, the per-block overhead is small relative to the matmul cost. Fragmentation is bounded by B; for generation lengths of 50-500 tokens, fragmentation runs 5-10%. **Why option C is wrong.** Page size 1 is technically possible; it just performs terribly. FlashAttention v2 supports block sizes well above 1024. **Why option D is wrong.** Warp size (32 threads) is a thread-grouping concept, not a memory-allocation constraint. Block sizes that are not multiples of warp size still work, just sometimes with less efficient threading.

Quick reference

TL;DR

Page size trades lookup overhead against fragmentation. Size 1 pays per-token lookups and breaks GPU coalescing; size 1024 wastes most of each block. 16 amortizes lookups while keeping fragmentation under 10%.

Memory aid

Easy to grasp

Picture an airline assigning seats by buying rows of seats in bulk and parcelling them out. If you buy one seat at a time, you spend all day at the ticket counter and pay a fee per seat. If you buy 1000 seats at once for every passenger, you only do one big purchase, but a passenger flying alone wastes 999 seats. Buying 16 at a time is a sweet spot: you only visit the ticket counter once per group of 16 passengers, and groups smaller than 16 only waste a handful of seats. vLLM picks 16 for the same reason.

Key concepts

PagedAttention KV cache vLLM serving engine Memory fragmentation

Detailed answer & concept explanation

~7 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Quick scan · 2–4 min

Everything important, quickly.

Walk the OS virtual-memory analogy, explain the lookup-overhead failure mode of B=1 and the fragmentation failure mode of B=1024, derive the throughput vs fragmentation tradeoff curve, connect to prefix-caching granularity, and close with workload-dependent tuning advice.

Code example

python

from vllm import LLM, SamplingParams

# Default block_size=16 is set inside vLLM and rarely overridden.
# Showing it explicitly for clarity.
llm = LLM(
    model="meta-llama/Llama-4-Maverick",
    block_size=16,  # PagedAttention block size in tokens
    gpu_memory_utilization=0.92,
)

# Smaller blocks (e.g., block_size=8) might marginally reduce fragmentation
# for short-form chat workloads but increase per-block overhead.
# Larger blocks (e.g., block_size=64) might help long-form generation but
# increase fragmentation on short requests.

sampling = SamplingParams(temperature=0.7, max_tokens=200)
outputs = llm.generate(["Hello, world!"], sampling)

Real-world examples

Real products, models, and research that use this idea.

vLLM defaults to block_size=16 for PagedAttention across all supported models (Llama 4 Maverick, Mistral, Qwen 3.5, etc.) as of the 2026 releases.
TensorRT-LLM uses similar block-based KV cache management with comparable block-size choices (default 64 in some configurations).
SGLang and similar serving engines also adopt block-based KV management with default block sizes in the 16-64 range.
DeepSeek V4's MLA reduces per-token KV bytes, which interacts with block-size choice but does not change the basic tradeoff.
vLLM's prefix caching feature relies on block-aligned prefix matching, which is another argument for keeping block size small.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QHow does the optimal block size change with FP8 KV cache versus FP16?

QWhy does PagedAttention eliminate external fragmentation but not internal fragmentation?

QHow does prefix caching interact with block size?

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Treating page size as 'just a config parameter' and missing that the two extremes lose to the middle for fundamentally different reasons: per-token overhead at one end, wasted memory at the other.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

References

Primary sources. Skim if you want the original framing.