Match each inference serving option to its strongest fit
Drag each answer to line up with its matching prompt
vLLM
Structured output and long shared prefix workloads (agents, JSON extraction)
TGI
Heterogeneous fleet (LLM plus reranker plus embedding) on shared NVIDIA GPUs
SGLang
Self-hosted Hugging Face models with first class quantization and Hub integration
Triton Inference Server
Autoscaled multi-model Python services with custom pre and post processing
Ray Serve
Self-hosted open weights with maximum tokens per second per GPU
Modal
Hosted frontier models under existing cloud account, IAM, and compliance
AWS Bedrock / GCP Vertex
Hosted serverless GPU for fast deploys without infra ops
Each serving option has one dominant differentiator (throughput, Hub integration, structured output, multi-framework, Python autoscale, serverless, or cloud-IAM), and the right pick depends on which one your workload
Imagine choosing kitchen equipment for different jobs. A high-volume pizza oven is great for pizza but bad at soup. A multi-burner range handles many dishes at once but is slower per dish. A food truck lets you serve anywhere without owning a kitchen, and a hotel catering contract means somebody else cooks entirely. Pick the wrong tool and your kitchen is either underused or overworked. LLM serving runtimes work the same way: vLLM is the high-volume pizza oven, Triton is the multi-burner range, Modal is the food truck, and Bedrock or Vertex is the hotel contract. None of them is universally best; each is best for the workload it was designed for.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
The inference serving stack used to be a one-name conversation (TF Serving for classical models, Triton for everyone else). By 2026 it split into roughly seven options that each own one differentiator, and matching workload to differentiator is the actual skill. Picking the wrong runtime cascades into wrong cost, wrong latency, and unnecessary ops burden.
This card walks through what each option uniquely offers, when the differentiator actually matters for a given workload, and the anti-patterns that show up when teams pick by reflex.
Throughput and ergonomics: vLLM, TGI, SGLang
Three options live in the open-source self-hosted LLM serving space and each owns a different angle.
vLLM owns raw throughput. PagedAttention treats the KV cache like virtual memory, eliminating the contiguous-slab waste that other runtimes pay when sequence lengths vary. Continuous batching schedules new requests into already-running batches at every decode step rather than waiting for the slowest sequence to finish. Combined, this delivers 2-4x more tokens per second-per-GPU than naive serving, which is the metric that drives self-hosted cost per million tokens.
TGI (Text Generation Inference) owns Hugging Face Hub integration and quantization ergonomics. A Rust core, one-line Hub model pulls, GPTQ, AWQ, and bitsandbytes wired in by default, plus AMD and Intel Gaudi backends in addition to NVIDIA. Throughput is within shouting distance of vLLM in 2026; the developer experience for HF-centric teams is the differentiator.
SGLang owns structured generation and shared-prefix workloads. RadixAttention reuses KV cache across requests with shared prefixes (the agentic pattern where every turn shares the long system prompt), and the SGLang DSL compiles structured-output constraints (JSON schemas, regex, choice-of-N) into efficient inference. Picked by teams running agent systems, function calling, or large-volume JSON extraction.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Anyscale runs Llama 4 Maverick on vLLM behind their hosted service, citing PagedAttention as the per-GPU throughput unlock.
- Hugging Face uses TGI to serve models on the Inference API and Endpoints product, leveraging the Rust runtime plus Hub integration.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow do you handle a workload that has both an LLM and a reranker plus embedding?
Either Triton if you want a single endpoint and shared GPUs, or vLLM for the LLM plus a separate small-model server (TEI for embeddings, Infinity for rerankers); the trade-off is operational weight versus per-model optimization.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Picking vLLM for every workload by reflex. It is the throughput leader for LLM-only self-hosted serving but it is not the right answer for heterogeneous fleets, structured-output workloads, or teams without GPU ops.
60 second bullets to scan on the way to the call.
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.