**Option A is correct.** bf16 (brain float 16) was designed at Google to match fp32's dynamic range. It uses 1 sign bit, 8 exponent bits, and 7 mantissa bits, the same exponent layout as fp32 but with a much smaller mantissa.
This means bf16 can represent the same scale of values fp32 can, roughly 1e-38 to 1e+38, just at coarser precision. The 7-bit mantissa gives about 2 to 3 decimal digits of precision per number, compared to fp32's roughly 7 digits.
fp16 takes the opposite trade-off. It uses 1 sign bit, 5 exponent bits, and 10 mantissa bits. More mantissa means better precision (about 3 to 4 decimal digits), but the smaller exponent caps the range at about 6.5e+4 on the high end and underflows around 6e-8 on the low end. Gradients during training can easily fall outside this range, which is why fp16 training notoriously requires loss scaling: multiply the loss by a constant before backward, then unscale the gradients before the optimizer step, to keep small gradients above the underflow threshold.
B is wrong because the formats really do have different bit layouts. C inverts reality: fp16 has the smaller exponent. D invents a missing sign bit, both formats are signed.
**Option A is correct, and the bit layout is the heart of the answer.**
fp32 uses 1 sign, 8 exponent, 23 mantissa. The 8-bit exponent gives dynamic range from roughly 1e-38 to 1e+38, far more than any single computation needs but the wide range means gradients can vary by many orders of magnitude without overflowing.
bf16 keeps the 8-bit exponent and truncates the mantissa to 7 bits (1-8-7). Dynamic range matches fp32 because the exponent layout is identical; precision drops to about 2 to 3 decimal digits. The explicit design goal is: keep the range, sacrifice precision.
fp16 takes the opposite trade-off (1-5-10). Precision is better than bf16 (about 3 to 4 decimal digits), but the 5-bit exponent caps the range at roughly 6.5e+4 on the high end and underflows around 6e-8 on the low end. Gradients flowing through softmax or other normalisers can easily underflow to zero in fp16.
This is why fp16 mixed-precision training requires loss scaling: multiply the loss by a large constant (e.g. 65536) before backward to lift small gradients into range, then divide gradients by the same constant before the optimizer step. The pattern works but adds complexity. bf16 avoids the issue because its range matches fp32's, which is why bf16 is the LLM training default on hardware that supports it.
The other options are wrong in specific ways. B claims the formats are bit-identical, which is false. C asserts fp16 has the larger exponent, which inverts the layout. D invents a missing sign bit; both formats are signed.
Quick reference
TL;DR
bf16 keeps fp32's 8-bit exponent so it matches fp32's dynamic range with less precision. fp16 has only a 5-bit exponent and overflows on values fp32 and bf16 handle fine.
Picture a 16-character licence plate. You can spend characters on the year of the car or on the unique serial. fp32 has 32 characters and lots of room for both. bf16 and fp16 only get 16 characters and have to choose. bf16 spends the same number on year as fp32 does, so it can stamp any year fp32 can, but it has fewer characters left for the serial so two plates can look the same. fp16 spends fewer characters on year and more on serial, so it tells two plates apart more easily but cannot stamp very old or very future years at all. In numerical training that 'year' is dynamic range and 'serial' is precision. bf16 trades precision to keep fp32's range. fp16 keeps more precision but loses range, which is why it needs extra scaffolding to train without overflowing.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Hardware without native bf16 support (e.g. older Volta or Pascal GPUs) bf16 ops fall back to slow paths or are unsupported entirely; fp16 with loss scaling is the practical choice on those generations.
Numerically sensitive inference where every bit of precision matters fp16's larger mantissa gives modestly better precision than bf16 for inference workloads where dynamic range is not an issue; specialised quantisation may be even better.
Workloads with mixed dtype requirements such as fp32 accumulation in attention The choice between bf16 and fp16 only applies to the bulk of the math; accumulators, optimizer states, and master weights stay in fp32 regardless and need separate handling.
Quick scan · 2–4 min
Everything important, quickly.
4 min: fp32 bit layout + bf16 keeps the 8-bit exponent + fp16 has only 5 exponent bits + why range matters more than precision for training + loss scaling for fp16 + hardware support and default choice for modern training.
Side-by-side
Aspect
fp32
bf16
fp16
Bit layout (sign / exp / mantissa)
1-8-23
1-8-7
1-5-10
Dynamic range (approx)
1e-38 to 1e+38
1e-38 to 1e+38
6e-8 to 6.5e+4
Decimal precision
~7 digits
~2-3 digits
~3-4 digits
Loss scaling needed for training
No
No
Yes
Hardware support
Universal
Ampere+, TPU v3+
Pascal and later
Real-world examples
Real products, models, and research that use this idea.
1Modern LLM fine-tuning recipes for Llama 4, Qwen 3.5, and DeepSeek V4 default to bf16 mixed precision on H100s and A100s because the dynamic range eliminates loss scaling.
2Hugging Face Transformers' Trainer accepts bf16=True and fp16=True flags; the bf16 flag is preferred when the hardware supports it, the fp16 flag triggers the loss-scaler scaffolding.
3PyTorch's torch.amp module exposes autocast(device_type='cuda', dtype=torch.bfloat16) for bf16 and a GradScaler companion for fp16 mixed-precision training.
4Google's TPU v3 and later support bf16 natively, which is the original hardware that drove the format's design; the same format was later adopted by NVIDIA from Ampere onward.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy does bf16's smaller mantissa not cause training problems for transformers?
Sign in or go Pro to see the approach for this follow-up.
QHow does dynamic loss scaling actually decide when to halve or double the scale factor?
Sign in or go Pro to see the approach for this follow-up.
QWhy do most fine-tuning recipes keep Adam state in fp32 even when training in bf16?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Treating bf16 and fp16 as interchangeable. They use different bit budgets, behave differently under gradient updates, and need different training scaffolding.