Click any words you think contain an error. Click again to unmark.
Tensor cores are fixed-function units: they need aligned shapes and supported dtypes to engage, and lower-precision dtypes hit strictly higher peaks. High GPU utilization does not mean they are saturated.
Think of a super-fast stamping press that stamps metal sheets, but only sheets cut to an exact size. Hand it an oddly-shaped sheet and it cannot use the fast press at all; it falls back to slow hand-cutting. So it is often cheaper to trim a small sheet up to the exact size, even though you waste a little metal, than to skip the press. The press also works faster on thinner sheets: a thin sheet stamps about twice as fast as a thick one. And here is the sneaky part. The factory's busy light can glow bright even when the fast press sits idle waiting for sheets to arrive on a slow conveyor. A bright busy light does not mean the expensive press is doing real work.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Tensor cores are the most misunderstood unit on a modern GPU, and this passage packs four of the classic misunderstandings into one paragraph. Underneath all four sits a single mental error: treating tensor cores as flexible general-purpose matrix machines that just go fast. They are not. They are fixed-function multiply-accumulate arrays that engage only under specific conditions and otherwise hand the work to slower general cores.
This deep dive does two things. First it walks the four explicit errors in the passage: arbitrary shapes do not hit peak, dtype support is fixed per generation, padding to alignment is usually a win, and precision changes the peak. Then it pivots to the error the passage hints at but a strong candidate should raise unprompted: that high GPU utilization means tensor cores are saturated.
That last point is where senior interviews live. A wall of green in nvidia-smi feels like proof the hardware is working hard. During autoregressive decode it is almost the opposite. The tensor cores are idle, waiting on HBM, while the utilization counter reads ninety-plus. By the end you should be able to explain why, and name the metric that tells the truth.
Why tensor cores are fixed function, not flexible
A tensor core is a hardware block that computes a small matrix-multiply-accumulate in a single instruction, far faster than assembling the same result from scalar fused-multiply-adds on CUDA cores. The speed comes precisely from being rigid. The datapath is wired for fixed fragment sizes and a fixed menu of operand dtypes. There is no general scheduler inside that adapts to whatever you hand it; there is a specific set of shapes and types the silicon was etched to accept.
That rigidity is the source of the first two passage errors. The claim that batched matmul hits peak at arbitrary shapes is false: the inner dimensions must align to the fragment tiling, which on Hopper means multiples of 16 for FP16 and BF16, and 32 for FP8 and INT8. A non-aligned shape cannot be tiled cleanly, so the kernel quietly takes a slower path. The word quietly matters here. There is no exception, no warning, and no log line. The kernel still produces correct numbers, just at a fraction of the rate, which is why these bugs survive code review and only surface when someone finally profiles.
The claim that tensor cores adapt to any dtype is equally wrong. Each architecture generation ships a specific list of supported types. Volta added FP16, Ampere added BF16 and INT8, Hopper added FP8, and Blackwell added FP4 and FP6. Anything outside the supported list for your generation runs on the general cores at a fraction of the throughput. The hardware does not negotiate; the compiler routes you to whatever path your shapes and types actually permit. A staff engineer internalizes this as a hard eligibility gate rather than a soft performance knob.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- NVIDIA Nsight Compute reports tensor-core active percentage and achieved FLOP per second separately from coarse GPU utilization, exactly because nvidia-smi util misleads on decode.
- vLLM and TensorRT-LLM use continuous batching to raise decode arithmetic intensity, since single-request decode leaves Hopper tensor cores starved on HBM bandwidth.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy does decode pin nvidia-smi utilization high while achieved FLOP per second stays tiny?
Utilization samples whether any kernel was resident in the window. A memory-bound decode kernel is resident the whole time but spends those cycles waiting on HBM reads, so the busy counter saturates while the multiply-accumulate units do almost no math.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Reading nvidia-smi at 95 percent and concluding tensor cores are saturated. During decode they sit idle waiting on HBM; the busy counter measures occupancy, not achieved FLOP per second.
60 second bullets to scan on the way to the call.
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.