TensorRT-LLM
Also known as: TRT-LLM
NVIDIA's compiled inference engine: highest throughput on NVIDIA GPUs, with longer build and iteration time than vLLM.
NVIDIA's high-performance LLM inference engine that compiles models to optimized CUDA kernels with kernel fusion, in-flight batching, and FP8/INT4 quantization. Targets maximum throughput on NVIDIA GPUs.
In practice
The throughput champion if you're stuck on NVIDIA. Senior infra interviews probe vLLM vs TensorRT-LLM trade-offs (iteration speed vs perf).
How it compares
TensorRT-LLM is a compiled engine optimized for max throughput; vLLM is a Python serving engine with faster iteration and PagedAttention.
Related topics
Questions that mention this term
- Match each production serving framework to its defining strength
- Spell out TensorRT-LLM and pinpoint its production differentiator
- Name TGI's maintainer and its niche among serving stacks
- Pick the serving changes TensorRT-LLM 0.18 (early 2026) actually shipped
- When would you choose vLLM, TensorRT-LLM, SGLang, or TGI for a production serving deployment?
- When NVIDIA Triton Inference Server is still the right pick over vLLM or TGI
Related terms
AI System Design
End-to-end design of production LLM systems: ingestion, retrieval, serving, eval, monitoring.
API LLM
An LLM accessed through a provider API: pay per token, get the frontier model, hand over ops.
Beam Search
Keep the K best partial sequences at each step; deterministic, breadth-first decoding.
FlashAttention
A memory-aware attention kernel that's 2-4x faster than vanilla, with identical math.
GGUF
Self-contained binary format for quantized LLMs; the standard for llama.cpp / Ollama / LM Studio.
Greedy Decoding
At each step, pick the single highest-probability token. Fast and deterministic, but often loops.