What does LoRA do, and why is it popular for fine-tuning?

**LoRA**, short for Low-Rank Adaptation, is a way to fine-tune a large model on a tiny budget. Instead of updating all of the model's weights (which can be 7 billion to 400 billion parameters), LoRA freezes the original weights and adds a small pair of trainable matrices next to specific layers. The trick is the shape of those matrices. For a layer with a weight matrix W of size d × d, LoRA introduces two smaller matrices A (size d × r) and B (size r × d), where r is a small rank like 8 or 16. The product BA has the same shape as W but a tiny parameter count, only 2 · d · r numbers instead of d · d. During training, only A and B receive gradient updates. At inference, the layer computes Wx + BAx. The new behavior is encoded entirely in the small delta. The wins are huge. Memory usage drops because most of the model isn't being trained. Training is faster. You can keep many fine-tunes around (one per task) and swap them on the same base model.

**LoRA** inserts a pair of low-rank matrices alongside specific weight matrices in the model and trains only those, leaving the original weights frozen. For a target weight matrix W of dimension d by d, LoRA learns A of shape d by r and B of shape r by d. The rank r is typically 8 to 64. The forward pass becomes: $$h = Wx + BAx$$ B is initialised to zero so the adapter starts as a no-op; gradients flow only through A and B. Trainable parameters drop from d squared to 2 d r, a 100x to 1000x reduction at typical sizes. The practical wins land in three places. Fine-tuning a 70B model fits in a single A100 or H100 instead of a multi-node cluster. Training time drops proportionally. Adapters are tiny (10 to 100 MB), so you can ship dozens per base model and swap at serving time. LoRA is not quantization (option A), not distillation (option C), and not data reordering (option D). Those describe different optimizations that compose with LoRA but are conceptually distinct.