Match each inference serving option to its strongest fit
Drag each answer to line up with its matching prompt
vLLM
Hosted frontier models under existing cloud account, IAM, and compliance
TGI
Self-hosted open weights with maximum tokens per second per GPU
SGLang
Hosted serverless GPU for fast deploys without infra ops
Triton Inference Server
Heterogeneous fleet (LLM plus reranker plus embedding) on shared NVIDIA GPUs
Ray Serve
Structured output and long shared prefix workloads (agents, JSON extraction)
Modal
Autoscaled multi-model Python services with custom pre and post processing
AWS Bedrock / Google Cloud Gemini Enterprise Agent Platform
Self-hosted Hugging Face models with first class quantization and Hub integration
Each serving option has one dominant differentiator (throughput, Hub integration, structured output, multi-framework, Python autoscale, serverless, or cloud-IAM); the right pick is the one your workload values most.
Imagine choosing kitchen equipment for different jobs. A high-volume pizza oven is great for pizza but bad at soup. A multi-burner range handles many dishes at once but is slower per dish. A food truck lets you serve anywhere without owning a kitchen, and a hotel catering contract means somebody else cooks entirely. Pick the wrong tool and your kitchen is either underused or overworked. LLM serving runtimes work the same way: vLLM is the high-volume pizza oven, Triton is the multi-burner range, Modal is the food truck, and Bedrock or Gemini Enterprise Agent Platform is the hotel contract. None of them is universally best; each is best for the workload it was designed for.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
The inference serving stack used to be a one-name conversation (TF Serving for classical models, Triton for everyone else). It has since split into roughly seven options that each own one differentiator, and matching workload to differentiator is the actual skill. Picking the wrong runtime cascades into wrong cost, wrong latency, and unnecessary ops burden.
Throughput and ergonomics: vLLM, TGI, SGLang
Three options live in the open-source self-hosted LLM serving space and each owns a different angle.
vLLM owns raw throughput. PagedAttention treats the KV cache like virtual memory, eliminating the contiguous-slab waste that other runtimes pay when sequence lengths vary. Continuous batching schedules new requests into already-running batches at every decode step rather than waiting for the slowest sequence to finish. Combined, this delivers 2-4x more tokens per second-per-GPU than naive serving, which is the metric that drives self-hosted cost per million tokens.
TGI (Text Generation Inference) owns Hugging Face Hub integration and quantization ergonomics. A Rust core, one-line Hub model pulls, GPTQ, AWQ, and bitsandbytes wired in by default, plus AMD and Intel Gaudi backends in addition to NVIDIA. The developer experience for HF-centric teams was the differentiator. TGI is now in maintenance mode, and Hugging Face recommends vLLM or SGLang instead.
SGLang owns structured generation and shared-prefix workloads. RadixAttention reuses KV cache across requests with shared prefixes (the agentic pattern where every turn shares the long system prompt), and the SGLang DSL compiles structured-output constraints (JSON schemas, regex, choice-of-N) into efficient inference. Picked by teams running agent systems, function calling, or large-volume JSON extraction.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Teams serving Llama 4 Maverick on vLLM commonly cite PagedAttention as the per-GPU throughput unlock.
- Hugging Face used TGI to serve models on its Inference API and Endpoints product, using the Rust runtime plus Hub integration. TGI is now in maintenance mode, and Endpoints recommends vLLM or SGLang.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow do you handle a workload that has both an LLM and a reranker plus embedding?
Either Triton if you want a single endpoint and shared GPUs, or vLLM for the LLM plus a separate small-model server (TEI for embeddings, Infinity for rerankers); the trade-off is operational weight versus per-model optimization.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Picking vLLM for every workload by reflex. It is the throughput leader for LLM-only self-hosted serving but it is not the right choice for heterogeneous fleets, structured-output workloads, or teams without GPU ops.
60 second bullets to scan on the way to the call.
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.