vLLM defaults to a page size of 16. Pick what makes 1 and 1024 worse choices.

MCQ·Medium·4.0 · 0·~1 min·Asked atCognizant Sourcegraph Uniphore·Relevant atNVIDIA

Attempt it

**PagedAttention** is the trick vLLM uses to manage the KV cache the way operating systems manage virtual memory. Instead of allocating one big contiguous chunk per request (the old way, which wastes a lot of memory when requests have different lengths), vLLM divides the GPU's KV memory into fixed-size **blocks** and assigns blocks to requests on demand. A per-request **block table** maps logical positions to physical blocks. The block size B is the parameter the question is asking about. vLLM defaults to B = 16. **Why B = 1 is bad.** Every single token attended to would require a separate block-table lookup. The attention kernel would do a memory-pointer chase per token, which is the worst possible pattern for GPU memory. GPUs reach peak bandwidth by reading contiguous chunks of memory in coalesced fashion; reading one token at a time defeats that completely. Latency explodes. **Why B = 1024 is bad.** A request that generates only 30 tokens still gets allocated one full 1024-token block. The remaining 994 slots in the block are wasted. This is **internal fragmentation**: memory you allocated but cannot use. With realistic generation-length distributions, B = 1024 wastes most of the KV cache, which is the exact thing PagedAttention exists to avoid. **Why B = 16 hits the sweet spot.** Sixteen tokens is enough for the GPU to do one coalesced read per block (no per-token overhead) while keeping the wasted-slot count per request under 16. For typical generation lengths (50-500 tokens), the fragmentation is under 10%. The choice between 8, 16, and 32 is more debatable; the choice between 1 and 16, or between 16 and 1024, is not.

**PagedAttention** is the OS-style memory management vLLM applies to the KV cache. The cache is partitioned into physical blocks of size B (tokens), and each request gets a logical block table mapping its positions to physical blocks on demand. The choice of B is a classic capacity vs overhead curve, and both extremes are bad for distinct reasons. **Why B = 1 fails.** Every position attended to costs a separate block-table lookup. The attention kernel becomes a sequence of indirect memory accesses, one per token, completely defeating GPU coalesced reads. Tensor-core matmul kernels expect contiguous K and V blocks; with B = 1, every read is a separate small fetch. Bandwidth utilization drops by an order of magnitude. **Why B = 1024 fails.** Internal fragmentation. A request that generates 30 tokens gets one block of 1024 slots, wasting 994. Across a fleet handling thousands of concurrent requests with diverse generation lengths, this wastes most of the KV cache. The entire motivation for PagedAttention is to reduce the wasted memory from naive contiguous-allocation schemes; B = 1024 reintroduces the same problem at coarser granularity. **Why B = 16 wins.** A 16-token block is enough for the GPU to do one coalesced load per block and amortize the page-table lookup over 16 attentions. Empirically, the per-block overhead is small relative to the matmul cost. Fragmentation is bounded by B; for generation lengths of 50-500 tokens, fragmentation runs 5-10%. **Why option C is wrong.** Page size 1 is technically possible; it just performs terribly. FlashAttention v2 supports block sizes well above 1024. **Why option D is wrong.** Warp size (32 threads) is a thread-grouping concept, not a memory-allocation constraint. Block sizes that are not multiples of warp size still work, just sometimes with less efficient threading.

TL;DR

Page size trades lookup overhead against fragmentation. Size 1 pays per-token lookups and breaks GPU coalescing; size 1024 wastes most of each block. 16 amortizes lookups while keeping fragmentation under 10%.

Memory aid

Easy to grasp

Picture an airline assigning seats by buying rows of seats in bulk and parcelling them out. If you buy one seat at a time, you spend all day at the ticket counter and pay a fee per seat. If you buy 1000 seats at once for every passenger, you only do one big purchase, but a passenger flying alone wastes 999 seats. Buying 16 at a time is a sweet spot: you only visit the ticket counter once per group of 16 passengers, and groups smaller than 16 only waste a handful of seats. vLLM picks 16 for the same reason.

Key concepts

PagedAttention KV cache vLLM serving engine Memory fragmentation

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

PagedAttention is the central memory-management innovation in vLLM that made high-throughput LLM serving possible at production scale. Before PagedAttention, KV-cache management for variable-length requests was a memory-wasting mess; afterwards, it became a tractable systems problem with well-understood tradeoffs.

The block-size parameter B is the most important tuning knob in the system, and its tradeoff is a textbook capacity vs overhead curve. This deep dive walks the OS virtual-memory analogy that motivates the design, explains the two failure modes at the extremes of B, derives the empirical sweet spot at B = 16, and surveys the workload-dependent tuning considerations that arise in production.

Mental model: PagedAttention is malloc for the KV cache, with B as the allocation granularity. Tiny B wastes time on bookkeeping; huge B wastes memory on partial blocks. The sweet spot is where neither cost dominates.

The OS virtual-memory analogy

Modern operating systems manage RAM by dividing it into fixed-size pages (typically 4 KB on x86) and using a per-process page table to map virtual addresses to physical pages. Processes can be allocated non-contiguous physical memory while seeing a contiguous virtual address space. This eliminates external fragmentation (free memory exists but cannot be allocated because no chunk is large enough) at the cost of internal fragmentation (each process's last page is typically partially used).

PagedAttention does the same thing for KV cache

The KV cache for a single token is a fixed-size object: 2 * d_head * n_kv_heads * 2 bytes for FP16. A request of length N tokens needs N of these objects per layer. The old approach was to pre-allocate max_length contiguous slots per request, which wastes (max_length - N) slots for every request shorter than max_length.

PagedAttention instead pre-allocates a global pool of fixed-size blocks in GPU memory (each block holding B tokens of KV data) and assigns blocks to requests on demand via a per-request block table. The block table maps logical positions (0 to N-1) to physical block IDs, exactly like a page table maps virtual addresses to physical pages.

What this eliminates

External fragmentation goes away completely. Any free block can serve any request, regardless of size. New requests get blocks as they generate, returned to the pool when they finish. The system can hold many more concurrent requests in the same KV memory budget than the old contiguous-allocation approach.

What remains: internal fragmentation

The last block of every request is partially used. A request that generates 30 tokens with B = 16 uses 2 blocks (32 slots), wasting 2 slots out of 32 = 6.25% fragmentation. A request that generates 30 tokens with B = 1024 uses 1 block (1024 slots), wasting 994 slots = 97% fragmentation.

Internal fragmentation is the unavoidable cost of fixed-block allocation, and the magnitude of the cost is directly proportional to B.

Why B = 1 destroys throughput

Why B = 1024 wastes the cache

Why B = 16 hits the sweet spot, and workload-dependent tuning

Situations where this technique stops working.

2–4 min · Everything important, quickly.

python

from vllm import LLM, SamplingParams

# Default block_size=16 is set inside vLLM and rarely overridden.
# Showing it explicitly for clarity.
llm = LLM(
    model="meta-llama/Llama-4-Maverick",
    block_size=16,  # PagedAttention block size in tokens
    gpu_memory_utilization=0.92,
)

# Smaller blocks (e.g., block_size=8) might marginally reduce fragmentation
# for short-form chat workloads but increase per-block overhead.
# Larger blocks (e.g., block_size=64) might help long-form generation but
# increase fragmentation on short requests.

sampling = SamplingParams(temperature=0.7, max_tokens=200)
outputs = llm.generate(["Hello, world!"], sampling)

Real products, models, and research that use this idea.

vLLM defaults to block_size=16 for PagedAttention across all supported models (Llama 4 Maverick, Mistral, Qwen 3.5, etc.) as of the 2026 releases.
TensorRT-LLM uses similar block-based KV cache management with comparable block-size choices (default 64 in some configurations).

What an interviewer would ask next. Try answering before peeking at the approach.

QHow does the optimal block size change with FP8 KV cache versus FP16?

FP8 halves the bytes per token, so a block of 16 FP8 tokens has half the bytes of a block of 16 FP16 tokens. GPU coalesced-read granularity is in bytes (typically 128B), so the optimal block size in tokens roughly doubles for FP8. In practice, vLLM and similar stacks expose block_size in tokens and let the user adjust if they care.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Treating page size as 'just a config parameter' and missing that the two extremes lose to the middle for fundamentally different reasons: per-token overhead at one end, wasted memory at the other.

60 second bullets to scan on the way to the call.

The OS virtual-memory analogy for PagedAttention's block-table design
Why page size 1 destroys GPU coalesced memory access and tensor-core utilization

Primary sources. Browse if you want the original framing.