TensorRT-LLM
Also known as: TRT-LLM
NVIDIA's compiled inference engine: highest throughput on NVIDIA GPUs, with longer build and iteration time than vLLM.
NVIDIA's high-performance LLM inference engine that compiles models to optimized CUDA kernels with kernel fusion, in-flight batching, and FP8/INT4 quantization. Targets maximum throughput on NVIDIA GPUs.
In practice
The throughput champion if you're stuck on NVIDIA. Senior infra interviews probe vLLM vs TensorRT-LLM trade-offs (iteration speed vs perf).
How it compares
TensorRT-LLM is a compiled engine optimized for max throughput; vLLM is a Python serving engine with faster iteration and PagedAttention.
Related topics
Related terms
KV Cache
Cache attention's K and V tensors per layer so each new token doesn't re-process every prior token.
Temperature
Sampling knob (low = focused, high = diverse), applied to the logits before softmax.
AI System Design
End-to-end design of production LLM systems: ingestion, retrieval, serving, eval, monitoring.
Quantization
Run the model at lower numerical precision to save memory and accelerate inference.
FlashAttention
A memory-aware attention kernel that's 2-4x faster than vanilla, with identical math.
Grouped-Query Attention (GQA)
Compromise between MHA and MQA: query heads share KV heads in groups, cutting KV cache by 4-8x.