Why is LoRA's B initialised to zero and A to Gaussian. What breaks if both are Gaussian?
LoRA initialises B to zero and A to a Kaiming/Gaussian distribution. Explain why this asymmetric scheme is chosen, what it guarantees at step 0, and what would go wrong if both matrices were Gaussian-initialised.
B = 0, A Gaussian guarantees BA = 0 at step 0 so the LoRA-adapted forward equals the base. Gaussian A ensures gradients flow into B once it starts moving. Both-Gaussian would inject random perturbation at start.
Imagine you have a finished oil painting (the trained base model) and you want to teach it one new detail without ruining what's already there. So you lay a clean sheet of tracing paper over the canvas and put a pencil and a brand-new dry stamp next to it. Before you start, the stamp has no ink on it, so pressing it leaves no mark, the painting underneath looks exactly the same. That's the trick: one tool starts empty so the picture is untouched on day one. The pencil sketches first, the stamp slowly picks up ink as you practice, and only then do tiny edits start appearing on top of the painting. If both tools started already inked, you'd press them down on day one and smudge the painting before you'd even decided what to draw.
Detailed answer & concept explanation~8 min readEverything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Everything important, quickly.
Write the two gradient equations, show why one-zero-one-Gaussian satisfies both 'clean start' and 'gradient flow' constraints, walk through the failure modes (both zero, both Gaussian), and mention PiSSA / LoftQ as deliberate variants that break BA = 0 for their own reasons.
| Init scheme | BA at step 0 | Gradient flow | Outcome |
|---|---|---|---|
| B = 0, A Gaussian | 0 | B receives gradient via A^T | Standard, recommended |
| A = 0, B Gaussian | 0 | A receives gradient via B^T | Equivalent variant |
| Both zero | 0 | Both zero: stalls | Training freezes |
| Both Gaussian | Random rank-r matrix | Both factors have gradient | Degraded start, unstable |
| PiSSA SVD init | Non-zero but meaningful | Both factors have gradient | Deliberate variant, faster convergence on some tasks |
Real products, models, and research that use this idea.
- HuggingFace PEFT: `lora_A` initialised Kaiming-uniform, `lora_B` initialised to zeros.
- Microsoft LoRA reference implementation uses the same pattern.
- PiSSA (Meng et al. 2024) deliberately breaks the BA = 0 convention with SVD-based init.
- LoftQ initialises B from quantisation error for improved QLoRA stability.
- DoRA preserves the BA = 0 property while adding a learned magnitude factor.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy specifically Kaiming initialisation for A and not uniform?
QHow does PiSSA's SVD-based init avoid the random-perturbation problem?
Don't say thisRed flags and common mistakes that signal junior thinking. Click to expand.
Red flags and common mistakes that signal junior thinking. Click to expand.
Initialising both B and A from Gaussian. The model now starts step 0 off-distribution due to the random BA perturbation, and training spends early steps un-learning that injection.
The night-before-the-interview bullets. Scan these on the way to the call.
Primary sources. Skim if you want the original framing.
Same topic, related formats. Practice these next.