FP4 uses 16 uniformly-spaced floating-point buckets; NF4 uses 16 non-uniform quantile buckets calibrated to a unit normal, so it captures more signal on bell-shaped weight tensors.
Imagine you have to pick exactly 16 height labels to describe everyone in a city. One approach spaces the labels uniformly from 4 feet to 7 feet, every 2.4 inches. That works, but most people cluster around 5'5" so half your labels are wasted on heights almost nobody has. The other approach looks at the actual height distribution first and places more labels in the crowded middle of the range and fewer in the rare extremes. The second approach loses less information about typical people because it puts its precision where the density is. NF4 is the second approach for transformer weights, which cluster around zero in a bell shape. FP4 is the first approach, spacing buckets evenly without knowing the data shape.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
FP4 versus NF4 is the textbook example of how quantisation quality depends not on the number of bits but on where those bits' representable values are placed. Both formats use exactly 4 bits per weight, so both have exactly 16 representable values. The difference between them is the placement of those 16 values on the number line, which determines how much information each quantised weight carries about its original floating-point value.
The answer in one line is that FP4 places its 16 values on a uniform-ish floating-point grid while NF4 places them at the quantile points of a standard normal distribution. Because pretrained transformer weights are roughly standard normal distributed after per-block scaling, NF4's bucket placement matches the data density and produces less quantisation error per weight. FP4's uniform placement wastes precision on the sparse tails and loses precision in the dense middle.
This deep dive walks through the codebook layouts of both formats, the empirical distribution of transformer weights that motivates NF4, the information-theoretic argument for quantile-based codebooks, and the practical edge cases where the assumption breaks down.
FP4: the uniform floating-point codebook
FP4 is a miniature floating-point format. Like fp16 or bf16, it splits its 4 bits across a sign bit, an exponent field, and a mantissa field. Different variants exist with different field widths (E2M1 has 2 exponent bits and 1 mantissa bit; E3M0 has 3 exponent bits and 0 mantissa bits), but the structural principle is the same: the 16 representable values are distributed on a logarithmic-uniform grid.
The logarithmic part means the spacing between consecutive values doubles as magnitude increases. Near zero, values are densely packed; far from zero, they spread out. This is the same principle that makes floating-point formats good at representing real numbers across many orders of magnitude.
The uniform part means the codebook makes no assumption about which values are common and which are rare. Every representable value is equally available for any weight that happens to land there, regardless of how the actual weight distribution looks.
For weight tensors whose distribution is log-uniform (rare in pretrained transformers) or close to it (also rare), FP4 is near-optimal. For weights that cluster strongly in a specific range, FP4 wastes representable values on the empty regions and provides too few values in the populated region. That mismatch is the quality gap NF4 closes.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Hugging Face TRL and Unsloth default to bnb_4bit_quant_type='nf4' for QLoRA fine-tuning Llama 4 Maverick on consumer GPUs.
- The QLoRA paper benchmarks NF4 versus FP4 across dozens of model families and shows NF4 wins consistently on perplexity for the same 4-bit budget.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy is NF4 only a small win over FP4 in practice rather than the dramatic win the information-theoretic argument suggests?
Consider block-wise scaling. Both formats rescale each block of 64 weights into the codebook range, so within-block distributions are normalised; the per-block scaling already absorbs most of the gain that NF4's prior would otherwise provide globally.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Assuming NF4 stores more bits per weight than FP4. Both are exactly 4 bits per value; the difference is where the 16 representable values are placed within the range.
60 second bullets to scan on the way to the call.
Both formats use 4 bits per weight with 16 representable codebook values
FP4 uses a uniform floating-point bucket layout with sign, exponent, mantissa fields
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.