FP4 vs NF4, pick the answer that captures the actual structural difference

MCQ·Easy·4.0 · 0·~1 min

Attempt it

**Both FP4 and NF4** are 4-bit number formats used to compress LLM weights for QLoRA-style fine-tuning. Four bits means each weight can only take one of 16 possible values. The two formats differ in what those 16 values are. **FP4** is a tiny floating-point format. It lays out its 16 values on a uniform exponent-mantissa grid, similar in spirit to fp16 but stripped down to four bits. The buckets are evenly spaced across the range in log-magnitude terms. **NF4** stands for 'Normal Float 4'. Its 16 values are placed at the quantile points of a standard normal distribution. The middle of the range has more closely-spaced buckets because that is where most pretrained transformer weights actually sit; the tails have sparser buckets because few weights live there. The key insight is that pretrained transformer weights are roughly normally distributed: zero-centered, bell-shaped, with most values close to zero. NF4's bucket placement matches this shape, so it spends its 16 buckets where the data density is, which means less quantization error per weight. FP4 ignores the data distribution and uses uniform buckets, wasting buckets on the empty tails and losing precision in the dense middle. In practice NF4 produces noticeably better quality than FP4 for the same 4-bit storage cost, which is why modern QLoRA recipes default to NF4.

**Both formats encode 4 bits per weight**, so they have exactly 16 representable values. The structural difference is where those 16 values sit on the number line. **FP4** uses a uniform floating-point grid. The 4 bits split into sign, exponent, and mantissa fields (typically 1-2-1 or 1-1-2 depending on variant), giving 16 evenly-spaced values across the representable range. It is essentially fp16's logic compressed to 4 bits, with the same uniform-density property. **NF4** (Normal Float 4) uses a non-uniform layout. Its 16 values are the quantiles of a standard normal distribution: the bucket centers are placed at the 1/16th, 3/16th, 5/16th, etc. points of the cumulative normal distribution. This means the buckets are tightly packed around zero, where the standard normal has its highest density, and sparsely spaced in the tails. The trick that makes NF4 work in practice is that pretrained transformer weights are empirically very close to standard normal, after per-block scaling. When you quantize a normal-distributed tensor with quantile buckets of that same distribution, the quantization error is roughly information-theoretically optimal for the given bit budget. FP4's uniform buckets waste precision on the sparse tails (where almost no weights live) and lose precision in the dense middle (where most weights live), so it has higher quantization error than NF4 for the same 4 bits per weight. The QLoRA paper benchmarks both formats and reports a consistent quality advantage for NF4 across model families, which is why every modern recipe defaults to `bnb_4bit_quant_type='nf4'`. FP4 remains supported for compatibility but rarely beats NF4 on any task.

Quick reference

TL;DR

FP4 uses 16 uniformly-spaced floating-point buckets; NF4 uses 16 non-uniform quantile buckets calibrated to a unit normal, so it captures more signal on bell-shaped weight tensors.

Memory aid

Easy to grasp

Imagine you have to pick exactly 16 height labels to describe everyone in a city. One approach spaces the labels uniformly from 4 feet to 7 feet, every 2.4 inches. That works, but most people cluster around 5'5" so half your labels are wasted on heights almost nobody has. The other approach looks at the actual height distribution first and places more labels in the crowded middle of the range and fewer in the rare extremes. The second approach loses less information about typical people because it puts its precision where the density is. NF4 is the second approach for transformer weights, which cluster around zero in a bell shape. FP4 is the first approach, spacing buckets evenly without knowing the data shape.

Key concepts

QLoRA NF4 quantization FP4 quantization

Detailed answer & concept explanation

~5 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Quick scan · 2–4 min

Everything important, quickly.

5 min: establish both formats are 4-bit, contrast uniform versus quantile codebook layouts, explain why the normal prior fits pretrained weights, then cover the block-scaling interaction and the FP4 default override.

Real-world examples

Real products, models, and research that use this idea.

Hugging Face TRL and Unsloth default to bnb_4bit_quant_type='nf4' for QLoRA fine-tuning Llama 4 Maverick on consumer GPUs.
The QLoRA paper benchmarks NF4 versus FP4 across dozens of model families and shows NF4 wins consistently on perplexity for the same 4-bit budget.
DeepSeek V4 community QLoRA recipes use NF4 to fit the 70B base into 4xH100 setups with room for LoRA training overhead.
Qwen 3.5 fine-tunes on 24GB consumer cards rely on NF4 quantization of the frozen base to leave enough memory for activations and LoRA gradients.
bitsandbytes still exposes FP4 as an option for compatibility but its documentation recommends NF4 as the default for pretrained LLM weights.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy is NF4 only a small win over FP4 in practice rather than the dramatic win the information-theoretic argument suggests?

QIf transformer weights were uniformly distributed instead of normally distributed, would FP4 or NF4 win?

QHow does NF4's quality advantage interact with the choice of block size in block-wise quantization?

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Assuming NF4 stores more bits per weight than FP4. Both are exactly 4 bits per value; the difference is where the 16 representable values are placed within the range.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

References

Primary sources. Skim if you want the original framing.