Fill in the memory delta when a 7B weight tensor moves from fp32 to bf16

**Floating-point formats** decide how many bytes one model weight occupies in GPU memory. fp32, the original training format, packs each weight into 32 bits, which is 4 bytes. bf16, the brain-float-16 format favored on modern accelerators, packs each weight into 16 bits, which is 2 bytes. The numerical range is similar in both formats; bf16 simply spends fewer bits on precision. Now scale that per-weight cost up. A 7 billion parameter model in fp32 holds its weights in roughly 7e9 times 4 bytes, which is 28 GB. The same 7 billion weights in bf16 hold in 7e9 times 2 bytes, which is 14 GB. The savings from the format swap is the difference between the two: 28 minus 14 equals 14 GB freed. That freed memory matters because GPU VRAM is the scarce resource during fine-tuning. The weights are only one of several tensors that share the device, and trimming them in half leaves room for optimizer states, gradients, activations, and a larger batch.

**fp32** stores each parameter in 4 bytes. **bf16** stores each parameter in 2 bytes. The factor between the two formats is exactly two. Multiply by parameter count to get the tensor footprint: $$\text{bytes} = N_{\text{params}} \times \text{bytes per param}$$ For a 7 billion parameter weight tensor: - fp32: 7e9 times 4 bytes equals roughly 28 GB - bf16: 7e9 times 2 bytes equals roughly 14 GB - Saved: 14 GB The freed memory is exactly half the fp32 footprint because halving the per-element cost halves the whole tensor. That 14 GB is real headroom: it can absorb a longer sequence, a bigger micro-batch, optimizer states, or activation memory. A caveat worth saying out loud: this calculation covers WEIGHTS only. A full fine-tune in fp32 carries gradients of the same size plus Adam's two moment buffers, so total per-parameter memory often reaches 16 bytes. Switching weights and gradients to bf16 while keeping fp32 master weights, the standard mixed-precision recipe, saves more than the weight delta alone. The 14 GB number is the answer to the literal question, not the full budget story.