Pick the serving changes TensorRT-LLM 0.18 (early 2026) actually shipped

Multi-select·Medium·4.0 · 0·~1 min·Asked atNVIDIA

Attempt it

**TensorRT-LLM** is NVIDIA's production inference engine for large language models. The 0.18 release in early 2026 shipped three notable changes that map onto the two big industry currents of the moment. The first current is Blackwell-era hardware. NVIDIA's B200 and B300 GPUs added native FP4 tensor cores, which deliver another 2x throughput jump beyond FP8 if the kernels exist to use them. TRT-LLM 0.18 added those FP4 kernels along with the quantisation path to convert FP8 or FP16 weights to FP4 for serving. The second current is convergence of serving engines on a shared scheduling design. vLLM v1 popularised chunked prefill, where a long prompt is sliced and interleaved with decode steps for other requests in the same scheduling tick. TRT-LLM 0.18 added an executor mode that does the same thing, ending the era of long prefills blocking all decodes. The third change is tighter speculative decoding integration. Medusa (multi-head drafts) and EAGLE (draft trees) are now first-class options in the engine, with built-in draft-target verification so users do not have to hand-code the verify loop. The wrong options describe things TRT-LLM did not do: switch the frontend to Rust, ship AMD/Intel backends, or drop in-flight batching. Real engine releases ship incremental wins along existing currents, not strategic pivots.

Three real changes in TRT-LLM 0.18, two distractor patterns to recognise. **FP4 kernels for Blackwell.** B200 and B300 ship native FP4 tensor cores at roughly 9000 TFLOP/s peak, double the FP8 rate. Using them requires kernels that accept FP4 operands and a weight-conversion path from FP8 or FP16. 0.18 added both, plus calibration recipes that hold accuracy on standard benchmarks under 1 percent loss for most models. **Chunked prefill executor.** vLLM v1 demonstrated that interleaving prefill chunks with decode steps eliminates head-of-line blocking when a long prompt arrives during active serving. TRT-LLM's new executor mode does the same: long prompts are sliced into configurable chunks and scheduled alongside in-flight decodes, so P99 TPOT stops spiking when a 16k-token prefill lands in the queue. **Speculative decoding integration.** Medusa-style multi-head drafts and EAGLE-style draft trees are now configurable in the executor without hand-coding the verify loop. The engine handles draft generation, target verification, and accept/reject masking. This brings TRT-LLM into parity with vLLM and SGLang on the spec-decoding feature surface. The wrong options share a pattern: they propose strategic pivots that would not appear in an incremental version bump. A Rust rewrite, multi-vendor backends, or dropping in-flight batching are the kinds of changes that take quarters and ship behind feature flags, not 0.18.x point releases. In-flight batching is the centrepiece of the engine; deprecating it would gut the product.

Quick reference

TL;DR

TRT-LLM 0.18 added FP4 kernels for Blackwell, chunked prefill in the executor (vLLM parity), and tighter Medusa/EAGLE speculative-decoding integration.

Memory aid

Easy to grasp

Think of TensorRT-LLM as a high-performance race car. The 0.18 update did three things: it added a new fuel grade that only the latest engines can burn (FP4 on Blackwell), it learned to weave through traffic more smoothly so one slow car cannot block the whole highway (chunked prefill interleaving with decode), and it bolted on better autopilot helpers that anticipate the next few moves (tighter speculative decoding). The wrong options would have been like swapping the engine for a different brand, repainting it for a different racing league, or ripping out the most important feature it had. Real release notes are incremental: you add new fuel grades and smarter logic, you do not throw out the chassis.

Key concepts

TensorRT-LLM FP4 (Blackwell)Chunked prefill Speculative decoding

Detailed answer & concept explanation

~7 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Quick scan · 2–4 min

Everything important, quickly.

4 min: FP4 kernels for Blackwell + chunked prefill executor (vLLM-parity) + Medusa/EAGLE speculative-decoding integration + why each distractor describes a pivot NVIDIA would not ship in 0.18.

Real-world examples

Real products, models, and research that use this idea.

TensorRT-LLM 0.18 shipped FP4 kernels for Blackwell B200/B300, doubling matmul throughput over FP8 with NVFP4 per-block scale calibration.
The new executor mode in 0.18 mirrors vLLM v1's chunked-prefill design, fixing the head-of-line blocking problem long prefills caused in prior releases.
Medusa and EAGLE speculative-decoding integration in 0.18 means production deployments can enable spec decoding via config rather than custom plugins.
In-flight batching remains the production default in TRT-LLM, the same continuous batching pattern vLLM and SGLang use as their core scheduler.
SGLang and vLLM ship multi-vendor backends (AMD MI300, Intel Gaudi); TRT-LLM stays NVIDIA-only by design as part of NVIDIA's serving stack.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QWhat is NVFP4 and how does it differ from vanilla FP4?

QHow does TRT-LLM's chunked prefill compare to vLLM v1's implementation?

QWhy is in-flight batching not an obvious thing to deprecate?

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Picking distractors that propose dramatic discontinuities (Rust rewrite, multi-vendor backends, dropping in-flight batching). Real engine releases ship incremental wins along industry currents, not strategic pivots.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

References

Primary sources. Skim if you want the original framing.