Fill in the memory delta when a 7B weight tensor moves from fp32 to bf16
fp32 uses 4 bytes per weight, bf16 uses 2, so a 7B tensor drops from 28 GB to 14 GB, freeing about 14 GB.
Imagine every weight is a card you have to store in a filing cabinet. In the wide format, each card needs a four-slot envelope. In the narrow format, each card only needs a two-slot envelope. The card itself does not change shape, only the packaging around it. If you have seven billion cards and you trade every wide envelope for a narrow one, you cut your storage shelves exactly in half. Same number of cards, half the cabinet space. That is the deal you get when you swap a model's weights from the wide format to the narrow one, and on a seven billion card collection the freed shelf space comes out to about fourteen gigabytes.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Memory math is the most common knock-out question in fine-tuning interviews because it separates people who have read about formats from people who have actually moved a model onto a GPU. The bf16 swap is the workhorse memory optimization of 2026 training stacks, and the calculation behind it is two arithmetic steps wrapped around one number you should have memorized: 4 bytes for fp32, 2 bytes for bf16.
The scenario in the question is simple. A 7 billion parameter weight tensor sits in fp32 on the GPU. You switch it to bf16. How much VRAM did you free? The answer is half the original footprint, which works out to roughly 14 GB. The path to that answer is what the rest of this section unpacks, because the same calculation generalizes to any tensor in any format and forms the basis of every fine-tuning memory budget you will ever build.
The surrounding context matters too. Memory on a GPU is shared across weights, gradients, optimizer states, activations, and KV-cache during inference. A clean weight-delta calculation is the entry point, but a serious engineer follows it immediately with the same math for gradients and optimizer states, because those quantities can dwarf the weight footprint during training.
Bytes per element: the one number to memorize
Every floating-point format reserves a fixed number of bits per value, and dividing by 8 gives bytes per value. fp32 uses 32 bits, which is 4 bytes. bf16 uses 16 bits, which is 2 bytes. fp16 also uses 16 bits and 2 bytes. int8 uses 8 bits and 1 byte. The 4-bit formats NF4 and FP4 each pack two values per byte.
The bit count is the only thing that determines per-element memory. Within a single bit width, the choice between formats (bf16 vs fp16, NF4 vs FP4) is about numerical behavior, not memory. bf16 spends most of its 16 bits on the exponent to match fp32's dynamic range; fp16 spends more bits on the mantissa for higher precision but a narrower range. For memory math, both cost 2 bytes per value.
This is also the answer to why halving the bit width halves the memory exactly. A weight tensor of N elements in any format costs N times its bytes-per-element. The total is linear in element count and linear in bytes per element, so a 2x reduction in bytes per element produces a 2x reduction in tensor footprint, no matter how many elements you have.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
import torch
n_params = 7_000_000_000
bytes_per_fp32 = 4
bytes_per_bf16 = 2
fp32_gb = n_params * bytes_per_fp32 / 1024**3
bf16_gb = n_params * bytes_per_bf16 / 1024**3
saved_gb = fp32_gb - bf16_gb
print(f"fp32 weights: {fp32_gb:.1f} GB")
print(f"bf16 weights: {bf16_gb:.1f} GB")
print(f"freed by swap: {saved_gb:.1f} GB")
# Verify with a real tensor
weights_bf16 = torch.empty(n_params // 1_000_000, dtype=torch.bfloat16)
weights_fp32 = weights_bf16.to(torch.float32)
assert weights_fp32.element_size() == 2 * weights_bf16.element_size()| Format | Bits | Bytes per param | 7B tensor size |
|---|---|---|---|
| fp32 | 32 | 4 | ~28 GB |
| bf16 | 16 | 2 | ~14 GB |
| fp16 | 16 | 2 | ~14 GB |
| int8 | 8 | 1 | ~7 GB |
| nf4 | 4 | 0.5 | ~3.5 GB |
Real products, models, and research that use this idea.
- Llama 4 Maverick checkpoints ship as bf16 weights on Hugging Face, halving the disk and VRAM footprint relative to an fp32 release.
- DeepSeek V4 base weights are distributed in bf16 because every modern Hopper and Blackwell GPU executes bf16 matmul natively at full throughput.
What an interviewer would ask next. Try answering before peeking at the approach.
QIf the model also needs gradients and Adam optimizer states, what is the full per-parameter memory in pure fp32 versus mixed-precision?
Pure fp32 with Adam is about 16 bytes per parameter: 4 weights, 4 gradients, 8 for the two moment buffers. Mixed-precision keeps bf16 weights (2) and bf16 gradients (2) but adds an fp32 master copy of the weights (4) plus fp32 first and second moments (8), landing near 16 bytes total per parameter with the activation footprint cut roughly in half.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Forgetting that the delta is the SAVINGS, not the new total. Reporting 28 GB or 16 GB instead of 14 GB usually means confusing the old size with the freed amount.
60 second bullets to scan on the way to the call.
Bytes per parameter in fp32 versus bf16
Why bits divide by eight to get bytes
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.