Spot the errors in this tensor-core utilization claim

Attempt it

Click any words you think contain an error. Click again to unmark.

Mark at least one word to submit.

Quick reference

TL;DR

Tensor cores are fixed-function units: they need aligned shapes and supported dtypes to engage, and lower-precision dtypes hit strictly higher peaks. High GPU utilization does not mean they are saturated.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Think of a super-fast stamping press that stamps metal sheets, but only sheets cut to an exact size. Hand it an oddly-shaped sheet and it cannot use the fast press at all; it falls back to slow hand-cutting. So it is often cheaper to trim a small sheet up to the exact size, even though you waste a little metal, than to skip the press. The press also works faster on thinner sheets: a thin sheet stamps about twice as fast as a thick one. And here is the sneaky part. The factory's busy light can glow bright even when the fast press sits idle waiting for sheets to arrive on a slow conveyor. A bright busy light does not mean the expensive press is doing real work.

Key concepts

Tensor cores Model FLOP utilization Memory-bound decode Low-precision dtypes

Detailed answer & concept explanation

~8 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Sign in for 5 free unlocks/day or go Pro for unlimited. See Pro pricing.

Quick scan · 2–4 min

Everything important, quickly.

4 min: spot the four claims (shape alignment, dtype menu, padding cost, precision peak) then pivot to the utilization trap, why decode is memory bound, and how MFU exposes idle tensor cores.

Real-world examples

Real products, models, and research that use this idea.

NVIDIA Nsight Compute reports tensor-core active percentage and achieved FLOP per second separately from coarse GPU utilization, exactly because nvidia-smi util misleads on decode.
vLLM and TensorRT-LLM use continuous batching to raise decode arithmetic intensity, since single-request decode leaves Hopper tensor cores starved on HBM bandwidth.
DeepSeek V4 and Llama 4 ship FP8 inference paths on Hopper and Blackwell to roughly double tensor-core peak versus BF16 on aligned matmuls.
PyTorch and cuBLAS pad GEMM dimensions up to tensor-core-friendly multiples automatically, because the padded fast path beats the unpadded fallback.
Training reports for GPT-5.5-class runs quote MFU near 40 to 50 percent, while single-stream decode on the same hardware sits in the low single digits despite high reported utilization.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy does decode pin nvidia-smi utilization high while achieved FLOP per second stays tiny?

Sign in or go Pro to see the approach for this follow-up.

QHow do you actually raise decode MFU on a starved Hopper GPU?

Sign in or go Pro to see the approach for this follow-up.

QWhy does INT8 roughly double FP16 tensor-core peak on the same silicon?

Sign in or go Pro to see the approach for this follow-up.

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Reading nvidia-smi at 95 percent and concluding tensor cores are saturated. During decode they sit idle waiting on HBM; the busy counter measures occupancy, not achieved FLOP per second.

Sign in or go Pro to see all red flags and common mistakes. See Pro pricing.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.