Mixed Precision

Which property explains bf16 preference over fp16 in frontier pretraining?

What is a tensor core and why does it dominate matmul throughput on GPUs?

Pick the most likely cause when training loss drops for 1k steps, plateaus for 2k, then drops again at step 3k

Same LoRA recipe on H100 vs A100: what precision should each team default to?

What does bnb_4bit_compute_dtype actually control in a QLoRA config?

bf16 vs fp16: which one preserves fp32's exponent range?

Adam optimizer state per parameter: fill in the moment buffers and bytes

A DPO run spikes to NaN around step 400 after a clean start: explain the most likely cause

Fill in the memory delta when a 7B weight tensor moves from fp32 to bf16

Explain bf16's advantage over fp16 and what gradient checkpointing trades

Why does fp16 fine-tuning need dynamic loss scaling but bf16 doesn't?

Fill in the full FT memory contributions for a 7B model

Where in attention is FP32 still used, and what breaks if you push everything to FP16/BF16?