Both A100 and H100 default to bf16; it hits tensor cores, matches fp32 dynamic range, and needs no loss scaler. fp8 is an opt-in only after numerics are validated.
Picture two photocopiers, an older one and a newer one. Both can copy at the same reliable mid-quality setting that almost never jams. The newer one also has an experimental ultra-fast mode that sometimes smudges if the paper is unusual. The safe default for both is the reliable mid-quality setting, and the experimental mode on the new copier is something you only enable after you have run a stack through cleanly and checked every page. Picking the experimental mode just because the new copier supports it would mean troubleshooting jams instead of getting work done. The same logic holds for precision on training hardware: pick what works on both, switch up only after you have a stable baseline.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Precision choices on training hardware look like a hardware-spec question and are really an operational-stability question. The right answer for a default LoRA recipe is the precision that hits tensor cores, runs with the fewest auxiliary subsystems, and works the same on both Ampere and Hopper. That precision is bf16.
The distractors in the question each represent a real temptation. fp16 was the default for years on Ampere before bf16 became universal in frameworks. fp8 on Hopper is genuinely faster on the right workload. fp32 is the safest fallback for debugging. None of them is the default in 2026; bf16 wins on the boring criteria that matter in production.
This deep dive walks through what bf16 actually is at the bit level, what fp16 buys and costs, why fp8 on Hopper is a careful opt-in rather than a free upgrade, and how to validate a precision switch when you do choose to move off bf16.
What bf16 actually is, and why it sidesteps loss scaling
bf16 (brain floating point 16) is a 16-bit format with one sign bit, an 8-bit exponent, and a 7-bit mantissa. The exponent width matters more than the total bit count: bf16 keeps the same 8-bit exponent as fp32, so its dynamic range is identical. The smallest representable positive value is the same; the largest is the same. Only the mantissa is coarser.
fp16 chose differently: 1 sign bit, 5-bit exponent, 10-bit mantissa. The exponent shrinkage limits the smallest representable value to around 2^-14 in normal range. Gradient magnitudes during training routinely run smaller than that, especially deep in a network, so without intervention they flush to zero and the optimizer sees no signal.
Dynamic loss scaling fixes fp16 by multiplying the loss by a large factor before backprop, shifting the gradient distribution up into fp16's representable range, then dividing the gradients by the same factor before the optimizer step. The framework adjusts the scaler dynamically: if any gradient overflows to inf the scaler halves and the step is skipped; after a streak of clean steps it doubles. It works, but it is a feedback loop with its own failure modes.
bf16 does not need any of this. Same dynamic range as fp32 means no underflow, which means no scaler. The mantissa is coarser, so individual values are less precise, but the impact on training quality is negligible for the kinds of computations a LoRA recipe runs.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
| Precision | A100 support | H100 support | Operational notes |
|---|---|---|---|
| fp32 | Yes (no tensor cores) | Yes (no tensor cores) | Stable but ~2x slower; only for debugging |
| fp16 | Tensor cores | Tensor cores | Needs dynamic loss scaling; narrow dynamic range |
| bf16 | Tensor cores | Tensor cores | Matches fp32 range; no loss scaler; safe default |
| fp8 (E4M3 / E5M2) | No native support | Tensor cores | Per-tensor scaling, outlier handling; opt-in only |
Real products, models, and research that use this idea.
- Hugging Face TRL and Transformers default the SFTTrainer to bf16 on both A100 and H100 SKUs, with fp8 gated behind explicit Transformer Engine integration.
- Unsloth and Axolotl LoRA recipes ship bf16 as the standard precision flag across open models like Llama 4 Maverick, Qwen 3.5, and DeepSeek V4.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy does bf16 avoid the dynamic loss scaler that fp16 needs?
Compare exponent widths. bf16 has the same 8-bit exponent as fp32, so the dynamic range is the same and small gradients do not flush to zero. fp16 has a 5-bit exponent and a much smaller minimum representable value, so without scaling the gradient distribution clips at the bottom.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Reaching for fp16 on A100 or fp8 on H100 as the obvious default. fp16 needs loss scaling and fp8 needs per-tensor calibration; bf16 sidesteps both on either chip.
60 second bullets to scan on the way to the call.
Why bf16 needs no loss scaler
Where fp16 underflows and how dynamic loss scaling fixes it
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.