LoRA init: why B = 0 and A ~ Gaussian (not both Gaussian)?

Attempt it

Quick reference

TL;DR

Set B = 0 so the LoRA update BA = 0 at step 0 and training starts exactly at the pretrained weights. Gaussian A keeps the gradient on A well-scaled once B moves.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Picture a skilled cook you want to teach a new dish. You want them to start cooking exactly the way they already do, then change things slowly, not throw the whole pot into chaos on the first try. Two helpers stand behind the cook, and the cook only changes the recipe when both helpers push together. So you tell one helper to push with zero force at the start. Because they multiply their efforts, nothing changes at all on the first try, and the dish tastes exactly like the cook's original. The other helper, though, is already leaning in a little, in a random direction. That tiny lean is what gives the first helper something to react to, so the very next try, real change begins. Start at the known-good point, then move on purpose.

Key concepts

LoRA adapters Weight initialization Parameter-efficient fine-tuning

Quick scan · 2–4 min

Everything important, quickly.

3 min: state the BA = 0 zero-perturbation start, explain why one factor is zeroed and the other Gaussian, debunk the Adam and rank distractors, then mention PiSSA and LoRA+ variants.

Side-by-side

Init choice	BA at step 0	Consequence
B = 0, A ~ Gaussian (standard)	Zero	Starts at base weights; gradient on B is well-scaled by random A
A = 0, B ~ Gaussian (mirror)	Zero	Also valid; only one factor needs to be zero
Both Gaussian	Random rank-r	Injects a random perturbation at step 0; degrades the starting point
Both zero	Zero	Degenerate gradient; adapter can stall and fail to learn

Real-world examples

Real products, models, and research that use this idea.

Hugging Face PEFT initialises LoRA B to zeros and A from a Kaiming-uniform draw, so every adapter trained on Llama 4 or Mistral starts at the base weights.
Unsloth and Axolotl, the popular 2026 LoRA training stacks, inherit this zero-B convention and expose an init_lora_weights flag built on it.
PiSSA and OLoRA are 2024-2025 variants that replace the zero-B start with SVD-based initialisation from the base weight to converge faster.
vLLM and S-LoRA multi-adapter serving rely on BA = 0 meaning a freshly attached untrained adapter is a no-op until it is trained.
DeepSeek V4 and Gemini 3.1 Pro adapter fine-tunes use the same asymmetric init in their published PEFT recipes.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QWalk through the step-0 gradients for A and B and show why zeroing both stalls learning.

Sign in or go Pro to see the approach for this follow-up.

QHow does PiSSA's initialisation differ, and why can it converge faster than the zero-B start?

Sign in or go Pro to see the approach for this follow-up.

QWhy does LoRA+ assign B a higher learning rate than A, and does that interact with the zero init?

Sign in or go Pro to see the approach for this follow-up.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.