LoRA initialises B to zero and A to a Kaiming/Gaussian distribution. Explain why this asymmetric scheme is chosen, what it guarantees at step 0, and what would go wrong if both matrices were Gaussian-initialised.
Free · 2 AI evals / day
**LoRA**, Low-Rank Adaptation, adds a small trainable update to a frozen base weight: `W = W_0 + (α/r) · B · A`. Before any training step, you want the adapted model to behave EXACTLY like the original. Otherwise, you've already nudged a known-good model into a worse one before learning anything.
Mathematically, that means `BA = 0` at step 0. A matrix product is zero whenever either factor is zero, so the easy fix is to set B to all zeros. Then BA = 0 no matter what A holds.
Why not zero both? Because then no gradient flows. The gradient into B is proportional to A, and the gradient into A is proportional to B. If both are zero, both updates are zero and training freezes.
So the recipe is asymmetric: one factor zero (clean start), the other Gaussian (so once B receives a small gradient via A, it can actually move). Convention is `B = 0, A ~ Gaussian`, but `A = 0, B ~ Gaussian` works identically. The asymmetry itself is the point.
LoRA's init has to satisfy two constraints at once, and they pull in opposite directions.
First, the adapted forward at step 0 should equal the base forward, same model, no perturbation. With `W = W_0 + (α/r) · BA`, that means `BA = 0` at step 0. The cleanest way to get there is to zero out one factor. People pick B, but zeroing A is mathematically equivalent.
Second, gradients have to actually flow once training starts. The gradient into B looks like `∂L/∂B ∝ A^T`, and the gradient into A looks like `∂L/∂A ∝ B^T`. If both were zero, both gradients vanish and the run just stalls.
So you need exactly ONE factor zero, for the clean start, and the OTHER non-zero, Kaiming or Gaussian, so that the zero factor immediately gets a non-degenerate gradient.
If you Gaussian-init BOTH, BA at step 0 is a random rank-r matrix glued onto W_0. You're effectively starting fine-tuning from a degraded checkpoint and burning the first few hundred steps un-doing that injection. With aggressive α/r or high rank, it can even blow up into NaN losses.
The trade-off worth flagging: variants like PiSSA deliberately break `BA = 0` and init from the top-r SVD of W_0, claiming faster convergence. But for vanilla LoRA, asymmetric zero-init is the default that does the real work.
Quick reference
TL;DR
B = 0, A Gaussian guarantees BA = 0 at step 0 so the LoRA-adapted forward equals the base. Gaussian A ensures gradients flow into B once it starts moving. Both-Gaussian would inject random perturbation at start.
Imagine you have a finished oil painting (the trained base model) and you want to teach it one new detail without ruining what's already there. So you lay a clean sheet of tracing paper over the canvas and put a pencil and a brand-new dry stamp next to it. Before you start, the stamp has no ink on it, so pressing it leaves no mark, the painting underneath looks exactly the same. That's the trick: one tool starts empty so the picture is untouched on day one. The pencil sketches first, the stamp slowly picks up ink as you practice, and only then do tiny edits start appearing on top of the painting. If both tools started already inked, you'd press them down on day one and smudge the painting before you'd even decided what to draw.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
You want a non-zero starting delta in a meaningful subspace Vanilla asymmetric init forces `BA = 0`; PiSSA-style SVD init or LoftQ-style quantization-error init give faster convergence when the base weight has dominant low-rank structure you want to start near.
Adapter variants that modify the BA decomposition (DoRA, VeRA) DoRA factors weight into magnitude and direction with its own init recipe; VeRA shares frozen random A/B across layers with only learned scalars. Copying vanilla LoRA's init breaks their parameterization assumptions.
QLoRA with aggressive 4-bit quantization of the base weight W₀ The quantized base already differs from the original; starting with `BA = 0` means step 0 sits on the quantized manifold, not the original. LoftQ explicitly initializes `BA` to absorb that quantization error.
Re-using a pre-trained LoRA adapter for continued training Re-zeroing B throws away learned signal. Resume from the saved A/B checkpoint; the zero-init contract only applies to a fresh adapter on top of an unmodified base.
Quick scan · 2–4 min
Everything important, quickly.
Write the two gradient equations, show why one-zero-one-Gaussian satisfies both 'clean start' and 'gradient flow' constraints, walk through the failure modes (both zero, both Gaussian), and mention PiSSA / LoftQ as deliberate variants that break BA = 0 for their own reasons.
Side-by-side
Init scheme
BA at step 0
Gradient flow
Outcome
B = 0, A Gaussian
0
B receives gradient via A^T
Standard, recommended
A = 0, B Gaussian
0
A receives gradient via B^T
Equivalent variant
Both zero
0
Both zero: stalls
Training freezes
Both Gaussian
Random rank-r matrix
Both factors have gradient
Degraded start, unstable
PiSSA SVD init
Non-zero but meaningful
Both factors have gradient
Deliberate variant, faster convergence on some tasks
Real-world examples
Real products, models, and research that use this idea.
1HuggingFace PEFT: `lora_A` initialised Kaiming-uniform, `lora_B` initialised to zeros.
2Microsoft LoRA reference implementation uses the same pattern.
3PiSSA (Meng et al. 2024) deliberately breaks the BA = 0 convention with SVD-based init.
4LoftQ initialises B from quantisation error for improved QLoRA stability.
5DoRA preserves the BA = 0 property while adding a learned magnitude factor.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy specifically Kaiming initialisation for A and not uniform?
Sign in or go Pro to see the approach for this follow-up.
QHow does PiSSA's SVD-based init avoid the random-perturbation problem?
Sign in or go Pro to see the approach for this follow-up.
QDoes the asymmetric init interact with LoRA+'s differential LR?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Initialising both B and A from Gaussian. The model now starts step 0 off-distribution due to the random BA perturbation, and training spends early steps un-learning that injection.