When NVIDIA Triton Inference Server is still the right pick over vLLM or TGI

Triton Inference Server is NVIDIA's general-purpose model server. It predates the LLM-specialized runtimes (vLLM, TGI, SGLang) and was originally designed for the broader inference market: vision models, classical ML, embedding models, recommendation systems. What makes Triton distinctive is **multi-framework, multi-model loading**. The same server instance can host PyTorch, TensorFlow, ONNX, TensorRT, and custom-backend models side by side. They share GPUs, share the request queue, and present one HTTP and gRPC endpoint to clients. For a team running an LLM plus an embedding model plus a reranker plus a vision classifier, Triton can serve all of it from one deployment. The **TensorRT-LLM backend** is the other angle. When the team needs maximum NVIDIA-blessed throughput for an LLM specifically, TensorRT-LLM compiled models running under Triton are competitive with vLLM and sometimes lead on specific configurations. The cost is **operational weight**. Triton's configuration model is heavier than vLLM's: model repositories, config files per model, scheduler tuning, version policies. For an LLM-only team, all of that is overhead with no corresponding benefit; vLLM gives more throughput per ops-hour for that workload. The practical guide is: heterogeneous model fleet, or TensorRT-LLM specifically, then Triton; LLM-only, then vLLM or TGI.

Triton is older than the LLM-specialized runtimes and still owns one workload class they cannot serve: **heterogeneous model fleets on shared GPUs behind one endpoint**. **The multi-framework story.** Triton loads PyTorch, TensorFlow, ONNX, TensorRT, and custom backends side by side. A single Triton instance can serve a Llama 4 Maverick model, a BGE-base embedding model, a Cohere reranker (if quantized to ONNX), and a YOLO vision classifier on shared GPUs behind one HTTP and gRPC endpoint. vLLM and TGI can serve only the LLM; the rest would need separate servers. **The TensorRT-LLM angle.** When raw NVIDIA-blessed throughput on a specific LLM matters, TensorRT-LLM compiled models running under Triton are competitive with vLLM and sometimes lead on specific batch-size and sequence-length configurations. NVIDIA pours significant engineering into this backend, so it stays close to the state of the art. **The operational cost.** Triton's deployment model is heavier than vLLM's: model repository layouts, per-model configuration files, scheduler tuning (dynamic batching policies, model instance counts, ensemble pipelines). For a team running one LLM and nothing else, this is overhead without corresponding benefit. **The decision tree.** Heterogeneous fleet sharing GPUs behind one endpoint? **Triton.** TensorRT-LLM throughput is the deciding metric? **Triton.** LLM-only deployment optimizing for cost per token? **vLLM or TGI.** Mixed: vLLM for the LLM, Triton or a dedicated embedding server (TEI, Infinity) for the support models, accepting the multi-server operational cost as the trade. The interview point is recognizing that the LLM-specialized runtimes did not make Triton obsolete; they made it specialized to a different workload.