Part ofAI Product Manager·Week 2: RAG & Use CasesView roadmap →

What does LoRA do, and why is it popular for fine-tuning?

MCQ·Medium·4.7 · 156·~1 min·Asked atAnduril Rephrase Ai Swiggy·Relevant atCohere Databricks Fireworks Ai Meta

Attempt it

TL;DR

LoRA freezes the base weights and trains tiny rank-r matrices that act as a low-rank delta. Same quality at 1/100 to 1/1000 the trainable parameters.

Memory aid

Easy to grasp

Imagine a huge textbook you cannot edit. You want to teach it a new chapter without rewriting the book. Instead of editing every page, you stick a thin pack of sticky notes (the LoRA adapter) on certain pages. The book never changes, the new ideas live on the sticky notes, and when you read a page you read the original plus whatever is on the sticky. Training is fast because there are only a few thousand sticky notes, not a million pages. Whenever you want a different specialty, you swap in a different sticky-note pack on the same book.

Key concepts

Low-Rank Adaptation QLoRA Parameter-efficient fine-tuning

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

LoRA is the single most important technique in the parameter-efficient fine-tuning toolkit. Released in 2021, it took over the open-weight ecosystem because it solved a specific bottleneck: full fine-tuning of a 70B model needs hundreds of GB of GPU memory just for optimizer states, putting it out of reach of most teams.

The technique itself is small enough to fit on a napkin. The real subtlety is why it works so well, why the low-rank prior is justified, and how the dozens of variants since (QLoRA, DoRA, AdaLoRA, PiSSA) refine the same core idea. Each variant pushes on a different dimension of the same problem: how do we capture maximum behavioral change with minimum trainable parameters, minimum memory, and minimum quality loss against a full fine-tune.

This deep dive walks the mechanism, the math, the production stack, and the boundary conditions where you should not reach for LoRA at all. By the end you should be able to size a LoRA fine-tune for any model in your head, justify the rank choice, and name three reasons the technique might be the wrong call for a specific task.

The mechanism: a low-rank additive delta

Start with a frozen pre-trained weight matrix W of shape d by d. Plain fine-tuning would replace W with W + ΔW, learning ΔW of the same shape. LoRA replaces the full ΔW with a low-rank factorisation:

\Delta W = BA, \quad A \in \mathbb{R}^{r \times d}, \quad B \in \mathbb{R}^{d \times r}

The rank r is tiny, typically 8 to 64. The forward pass becomes h = Wx + BAx. During training only A and B receive gradients; W stays frozen.

B is initialised to zero and A to a small random Gaussian. With B=0, BA=0 at step 0, so the adapted model behaves exactly like the base. This avoids any warm-start drift and lets the adapter learn the delta cleanly from the first gradient step. The scaling factor alpha over r is applied to the adapter output to decouple effective learning rate from rank choice.

Why the low-rank prior works

Production serving and adapter routing

Variants and the QLoRA stack

Tuning rank, target modules, and the common pitfalls

Situations where this technique stops working.

2–4 min · Everything important, quickly.

python

from peft import LoraConfig, get_peft_model
from transformers import AutoModelForCausalLM

base = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-4-8B")

config = LoraConfig(
    r=16,                              # rank
    lora_alpha=32,                     # scaling factor
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj",
                    "gate_proj", "up_proj", "down_proj"],
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM",
)

model = get_peft_model(base, config)
model.print_trainable_parameters()
# trainable params: 41M || all params: 8.04B || trainable: 0.51%

Real products, models, and research that use this idea.

OpenAI's fine-tuning API for GPT-4o-mini uses LoRA-style adapters under the hood, exposing only the high-level training endpoint.
Together.ai serves dozens of customer LoRA adapters on the same Llama 3.1 base via per-request adapter routing.

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy is B initialised to zero in LoRA?

Initialise BA to zero so the adapter starts as an identity operation. A is random Gaussian; B starts at zero so the model behaves exactly like the base at step 0, and the adapter learns the delta from there. Avoids catastrophic warm-start drift.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Confusing LoRA with quantization or distillation. LoRA does not shrink the model, it adds a small trainable delta on top of frozen base weights.

60 second bullets to scan on the way to the call.

Why LoRA freezes the base and trains the delta
The BA factorisation and why rank r matters

Primary sources. Browse if you want the original framing.