Gradient Checkpointing

LoRA freezes the base weights: does that mean activation checkpointing on the base is unnecessary?

Adam stores two fp32 buffers per trainable parameter: what does that imply for LoRA vs full FT on a 7B?

Gradient checkpointing, what does it trade, and roughly by how much?

Explain bf16's advantage over fp16 and what gradient checkpointing trades