LoRA init: why B = 0 and A ~ Gaussian (not both Gaussian)?
Set B = 0 so the LoRA update BA = 0 at step 0 and training starts exactly at the pretrained weights. Gaussian A keeps the gradient on A well-scaled once B moves.
Picture a skilled cook you want to teach a new dish. You want them to start cooking exactly the way they already do, then change things slowly, not throw the whole pot into chaos on the first try. Two helpers stand behind the cook, and the cook only changes the recipe when both helpers push together. So you tell one helper to push with zero force at the start. Because they multiply their efforts, nothing changes at all on the first try, and the dish tastes exactly like the cook's original. The other helper, though, is already leaning in a little, in a random direction. That tiny lean is what gives the first helper something to react to, so the very next try, real change begins. Start at the known-good point, then move on purpose.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
LoRA freezes the pretrained weight matrix and learns a small additive update expressed as the product of two low-rank factors. The effective weight is the frozen base plus a scaled product of a down-projection and an up-projection. The whole appeal is that you train only those two small factors instead of the full matrix, cutting trainable parameters by a hundred to a thousand times.
This question targets a detail that looks trivial but is genuinely load-bearing: the initialisation is asymmetric. One factor starts at exactly zero, the other starts at small Gaussian random values. People often half-remember the rule as 'zero one of them' without being able to say why each half matters, and that is exactly the gap a good interviewer probes.
The answer has two coupled parts. The zero factor guarantees the update is exactly zero at the start, so training begins at the pretrained weights with no random shock. The Gaussian factor guarantees a usable gradient once the zero factor starts to move. Get either half wrong and you either perturb a perfectly good model or stall the optimiser. This deep dive works through the mechanism, the math, the distractors, and the modern variants.
The reparameterisation and what 'start at the base' means
LoRA replaces a weight update with a low-rank product. The forward pass uses the frozen base weight plus a scaled adapter term, where the adapter is the product of an up-projection and a down-projection.
The whole design goal at initialisation is that the adapted model should be indistinguishable from the frozen base model. That means the added term must be zero on step 0. If it were not, you would be starting the fine-tune from a randomly perturbed model that is strictly worse than the checkpoint you paid to pretrain.
The cheap way to force the product to zero is to zero one factor. With B set to all zeros, the product BA is the zero matrix regardless of A. So the effective weight equals the base weight, and the first forward pass reproduces the base model exactly. Training then moves outward from a known-good point rather than from noise.
Contrast this with initialising both factors at random. The product BA would then be a random matrix of rank r, and the effective weight would be the base weight plus a random shift. The model you start from is no longer the pretrained checkpoint, it is a damaged copy of it. The first few hundred steps are then wasted just undoing that random damage before any useful learning happens. The zero-product start sidesteps that entirely: there is nothing to undo, so the very first gradient step is productive.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
| Init choice | BA at step 0 | Consequence |
|---|---|---|
| B = 0, A ~ Gaussian (standard) | Zero | Starts at base weights; gradient on B is well-scaled by random A |
| A = 0, B ~ Gaussian (mirror) | Zero | Also valid; only one factor needs to be zero |
| Both Gaussian | Random rank-r | Injects a random perturbation at step 0; degrades the starting point |
| Both zero | Zero | Degenerate gradient; adapter can stall and fail to learn |
Real products, models, and research that use this idea.
- Hugging Face PEFT initialises LoRA B to zeros and A from a Kaiming-uniform draw, so every adapter trained on Llama 4 or Mistral starts at the base weights.
- Unsloth and Axolotl, the popular 2026 LoRA training stacks, inherit this zero-B convention and expose an init_lora_weights flag built on it.
What an interviewer would ask next. Try answering before peeking at the approach.
QWalk through the step-0 gradients for A and B and show why zeroing both stalls learning.
Differentiate the loss through W + (alpha/r) B A. The gradient on A scales with B and the gradient on B scales with A; zero both and the useful signal collapses on the first step.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Justifying only the zero on B and forgetting why A must be random, or inventing a reason like Adam skipping zeros or rank preservation. The real reason is a zero-perturbation start plus a usable gradient.
60 second bullets to scan on the way to the call.
Why BA must be zero at step 0
What zeroing exactly one factor achieves
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.