Explain why vLLM became the default serving runtime for self-hosted open-weights models

**Self-hosted LLM serving** means running the model on your own GPUs rather than calling a hosted provider. The choice of serving runtime determines how many tokens per second a given GPU can produce, which directly sets your cost per million tokens. **vLLM** is the open-source serving runtime that has become the default pick for self-hosted inference in 2026. It is used in production by enterprises running Llama 4, Mistral, Qwen, DeepSeek, and other open-weights models. Two architectural choices made it the default: First, **PagedAttention**. The KV cache (the stored attention keys and values for each generated token) was traditionally allocated as a contiguous slab per sequence, sized for the worst case. Most real sequences are far shorter than the max, so the GPU memory was mostly empty. PagedAttention treats the KV cache like operating-system virtual memory: store the KV in fixed-size pages, index through a page table, and only allocate the pages a sequence actually needs. Memory savings are typically 2-4x. Second, **continuous batching**. Naive batching waits for the entire batch to finish before starting the next one, so the GPU idles whenever any sequence in the batch is still generating. Continuous batching admits new requests into already-running batches at every decoding step. The GPU stays saturated, and tokens per second per GPU jumps significantly. Other runtimes (TGI, SGLang, Triton, Ray Serve) have caught up to varying degrees, but vLLM established the patterns and remains the default starting point.

**Two architectural choices.** PagedAttention and continuous batching, each addressing a different bottleneck in transformer serving. **PagedAttention** is the memory-side win. Standard attention implementations allocate the KV cache contiguously per sequence, sized for the maximum context length. In production, most sequences are far shorter than the maximum, so the GPU memory is mostly reserved but empty. PagedAttention borrows the operating-system virtual-memory model: the KV cache is split into fixed-size blocks (typically 16 tokens per block), each sequence gets a page table mapping its logical KV positions to physical blocks, and blocks are allocated only as the sequence grows. The memory savings are 2-4x in practice, which directly translates into 2-4x more concurrent sequences per GPU. For long-context workloads or workloads with bursty length distribution, the gain is even larger. **Continuous batching** is the time-side win. Naive (static) batching forms a batch at request arrival, runs the entire batch through the model until every sequence finishes, then moves to the next batch. The slowest sequence in a batch holds up the entire batch, common when generations have very different output lengths. Continuous batching uses an iteration-level scheduler that examines the running batch at every decoding step, evicts finished sequences, admits new sequences from the queue, and re-batches. The GPU never idles waiting for one long generation, and tokens per second per GPU jumps materially. **The combined effect** is tokens per second per GPU that is multiples higher than naive serving, which is the metric that drives cost per million tokens for self-hosted inference. That is why vLLM became the default: the cost per million tokens win is large enough to be the deciding factor for any team with non-trivial volume. **Other runtimes have caught up partially.** **TGI** (Hugging Face Text Generation Inference) implements continuous batching and has its own