LoRA init: why B = 0 and A ~ Gaussian (not both Gaussian)?
Set B = 0 so the LoRA update BA = 0 at step 0 and training starts exactly at the pretrained weights. Gaussian A keeps the gradient on A well-scaled once B moves.
Picture a skilled cook you want to teach a new dish. You want them to start cooking exactly the way they already do, then change things slowly, not throw the whole pot into chaos on the first try. Two helpers stand behind the cook, and the cook only changes the recipe when both helpers push together. So you tell one helper to push with zero force at the start. Because they multiply their efforts, nothing changes at all on the first try, and the dish tastes exactly like the cook's original. The other helper, though, is already leaning in a little, in a random direction. That tiny lean is what gives the first helper something to react to, so the very next try, real change begins. Start at the known-good point, then move on purpose.
Detailed answer & concept explanation~8 min readEverything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Everything important, quickly.
3 min: state the BA = 0 zero-perturbation start, explain why one factor is zeroed and the other Gaussian, debunk the Adam and rank distractors, then mention PiSSA and LoRA+ variants.
| Init choice | BA at step 0 | Consequence |
|---|---|---|
| B = 0, A ~ Gaussian (standard) | Zero | Starts at base weights; gradient on B is well-scaled by random A |
| A = 0, B ~ Gaussian (mirror) | Zero | Also valid; only one factor needs to be zero |
| Both Gaussian | Random rank-r | Injects a random perturbation at step 0; degrades the starting point |
| Both zero | Zero | Degenerate gradient; adapter can stall and fail to learn |
Real products, models, and research that use this idea.
- Hugging Face PEFT initialises LoRA B to zeros and A from a Kaiming-uniform draw, so every adapter trained on Llama 4 or Mistral starts at the base weights.
- Unsloth and Axolotl, the popular 2026 LoRA training stacks, inherit this zero-B convention and expose an init_lora_weights flag built on it.
- PiSSA and OLoRA are 2024-2025 variants that replace the zero-B start with SVD-based initialisation from the base weight to converge faster.
- vLLM and S-LoRA multi-adapter serving rely on BA = 0 meaning a freshly attached untrained adapter is a no-op until it is trained.
- DeepSeek V4 and Gemini 3.1 Pro adapter fine-tunes use the same asymmetric init in their published PEFT recipes.
What an interviewer would ask next. Try answering before peeking at the approach.
QWalk through the step-0 gradients for A and B and show why zeroing both stalls learning.
QHow does PiSSA's initialisation differ, and why can it converge faster than the zero-B start?
Don't say thisRed flags and common mistakes that signal junior thinking. Click to expand.
Red flags and common mistakes that signal junior thinking. Click to expand.
Justifying only the zero on B and forgetting why A must be random, or inventing a reason like Adam skipping zeros or rank preservation. The real reason is a zero-perturbation start plus a usable gradient.
The night-before-the-interview bullets. Scan these on the way to the call.
Primary sources. Skim if you want the original framing.
Same topic, related formats. Practice these next.