Memory Math

Adam stores two fp32 buffers per trainable parameter: what does that imply for LoRA vs full FT on a 7B?

Adam optimizer state per parameter: fill in the moment buffers and bytes

Fill in the memory delta when a 7B weight tensor moves from fp32 to bf16

How does QLoRA fit a 65B fine-tune onto a single 48GB GPU?

Estimate peak VRAM for a 7B full FT run in bf16 + Adam

Fill in the full FT memory contributions for a 7B model

Compute the KV cache size for a 70B model at 128k context (80 layers, 64 heads, head_dim 128, FP16).

Predict the KV cache size in GB for given model dimensions