What is the primary mechanism by which PagedAttention increases serving throughput?

Every request to an LLM server needs a **KV cache**, a chunk of GPU memory that grows by one slot per token generated. The hard part is you do not know in advance how long the answer will be. A naive server plays it safe and reserves one big contiguous block sized for the maximum possible length. That reservation is mostly wasted. If a request reserves space for 2048 tokens but generates only 200, the other 1848 slots sit blocked and unused. This is internal fragmentation, and across a busy server it can waste the majority of GPU memory. **PagedAttention** borrows the idea of virtual memory from operating systems. Instead of one big contiguous block, it splits the cache into small fixed-size blocks, say 16 tokens each. Each request gets a block table mapping its logical token positions to physical blocks scattered anywhere in memory. Now memory is handed out one small block at a time, only when a token actually needs it. The wasted reservation disappears. The freed memory lets the scheduler admit more concurrent requests, which is the whole point: a bigger batch amortizes each weight read across more tokens and throughput climbs sharply.

**PagedAttention** is a KV cache memory manager, not a change to attention arithmetic. The math each token computes is identical. What changes is how the cache is laid out in GPU memory. Naive serving reserves one contiguous KV region per request, sized for the maximum sequence length. Most requests finish far short of that. The reserved slots are internal fragmentation. Gaps between freed regions are external fragmentation. Together they can waste 60 to 80 percent of GPU memory. PagedAttention splits the cache into fixed-size blocks, typically 16 tokens. A per-request block table maps logical positions to physical blocks that can sit anywhere. Memory is allocated one block at a time, only as tokens are produced. Internal fragmentation drops to at most one partial block per request. External fragmentation vanishes because all blocks are interchangeable. The recovered memory is the lever. With far less waste, the scheduler admits many more concurrent requests, so effective batch size rises sharply. Decode is memory bandwidth bound. A larger batch amortizes each per-step weight read across more output tokens. That is the throughput win, and it also unlocks prefix sharing across requests.