Why is LoRA's B initialised to zero and A to Gaussian. What breaks if both are Gaussian?
LoRA initialises B to zero and A to a Kaiming/Gaussian distribution. Explain why this asymmetric scheme is chosen, what it guarantees at step 0, and what would go wrong if both matrices were Gaussian initialised.
B = 0, A Gaussian guarantees BA = 0 at step 0 so the LoRA-adapted forward equals the base. Gaussian A ensures gradients flow into B once it starts moving. Both-Gaussian would inject random perturbation at start.
Imagine you have a finished oil painting (the trained base model) and you want to teach it one new detail without ruining what's already there. So you lay a clean sheet of tracing paper over the canvas and put a pencil and a brand-new dry stamp next to it. Before you start, the stamp has no ink on it, so pressing it leaves no mark, the painting underneath looks exactly the same. That's the trick: one tool starts empty so the picture is untouched on day one. The pencil sketches first, the stamp slowly picks up ink as you practice, and only then do tiny edits start appearing on top of the painting. If both tools started already inked, you'd press them down on day one and smudge the painting before you'd even decided what to draw.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
LoRA freezes the pretrained weight matrix and learns a small additive update written as the product of two low-rank factors. The effective weight is the frozen base plus a scaled product of a down-projection and an up-projection. You train only those two small factors, which cuts trainable parameters by a hundred to a thousand times.
This question hinges on a detail that looks cosmetic but is genuinely load-bearing: the initialisation is asymmetric. One factor starts at exactly zero, the other starts at small Gaussian random values. Most people half-remember the rule as 'zero one of them' without being able to defend why each half is necessary.
The justification has two coupled parts. The zero factor makes the update exactly zero at step 0, so training begins precisely at the pretrained weights with no random shock. The Gaussian factor seeds a usable gradient so the zero factor can leave zero on the first update. Get either half wrong and you either perturb a perfectly good model or freeze the optimiser. This deep dive works through the mechanism, the math, the failure modes, and the modern variants that deliberately break the rule.
The reparameterisation and what 'start at the base' means
LoRA replaces a weight update with a low-rank product. The forward pass uses the frozen base weight plus a scaled adapter term, where the adapter is the product of an up-projection B and a down-projection A.
The whole design goal at initialisation is that the adapted model is indistinguishable from the frozen base. That means the added term must vanish on step 0, so the constraint is simply that the product BA equals the zero matrix at the start.
The cheapest way to force that is to zero exactly one factor. With B set to all zeros, the product BA is the zero matrix regardless of A. The zero matrix is an absorbing element under multiplication, so nothing A holds can change that. The effective weight equals the base weight, and the first forward pass reproduces the base model exactly.
That property is the reason the scheme exists. Training moves outward from a known-good checkpoint rather than from noise. The very first gradient step is productive because there is nothing random to undo before useful learning can begin.
This is also why the contract is phrased as exact identity, not approximate identity. You do not want a starting point near the base, you want the base itself. Any non-zero BA at step 0 means the fine-tune begins on a strictly worse model than the checkpoint you paid to pretrain. You then spend early gradient steps clawing back to where you already were.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
| Init scheme | BA at step 0 | Gradient flow | Outcome |
|---|---|---|---|
| B = 0, A Gaussian | 0 | B receives gradient via A^T | Standard, recommended |
| A = 0, B Gaussian | 0 | A receives gradient via B^T | Equivalent variant |
| Both zero | 0 | Both zero: stalls | Training freezes |
| Both Gaussian | Random rank-r matrix | Both factors have gradient | Degraded start, unstable |
| PiSSA SVD init | Non-zero but meaningful | Both factors have gradient | Deliberate variant, faster convergence on some tasks |
Real products, models, and research that use this idea.
- HuggingFace PEFT: `lora_A` initialised Kaiming-uniform, `lora_B` initialised to zeros.
- Microsoft LoRA reference implementation uses the same pattern.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy specifically Kaiming initialisation for A and not uniform?
Kaiming/He scaling (σ = sqrt(2 / fan_in)) ensures the variance of activations stays roughly constant across layers, which is the standard requirement for stable training of deep networks. Uniform without proper variance scaling would cause vanishing/exploding signals in deep models.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Initialising both B and A from Gaussian. The model now starts step 0 off-distribution due to the random BA perturbation, and training spends early steps un-learning that injection.
60 second bullets to scan on the way to the call.
Why BA = 0 must hold at step 0
Why one of A or B must be the zero factor
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.