LoRA+: why does B want a higher learning rate than A?
B starts at zero and its gradient is bottlenecked, so a single shared learning rate under-trains it. LoRA+ gives B a roughly 16x higher rate than A.
Picture two people pushing a heavy cart together. One person is already leaning into it, warmed up and moving. The other is standing still with arms at their sides, starting from a dead stop. If you tell both to push with the same effort, the one starting cold barely helps, because they first have to overcome being frozen in place. The cart drifts unevenly and crawls along. So a smart coach tells the cold starter to push much harder than the warmed-up one, roughly sixteen times as hard, until both are shoving the cart at the same real pace. Now their combined effort moves the cart smoothly and quickly, instead of one person doing almost everything while the other lags behind the whole way. Match the effort to who is behind.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
LoRA+ is a one-line change to how you set learning rates in LoRA, and it routinely buys faster convergence plus a small accuracy bump. The idea sounds almost too small to matter: instead of using one learning rate for the whole adapter, you give the B matrix a higher rate than the A matrix, with a ratio around sixteen. The interesting part is why that asymmetry is the right call, and why getting its direction backwards is such a common interview error.
The answer lives entirely in how LoRA is initialised and how gradients flow through a product of two matrices. Once you see that B starts at exactly zero while A starts as small Gaussian noise, the asymmetry in the gradients falls out, and the corrective learning-rate ratio follows directly. The factorised structure that makes LoRA cheap is the same structure that makes a single learning rate the wrong tool.
This question is a favourite at the senior and staff level precisely because the surface reasoning and the correct reasoning point in opposite directions. A candidate who counts parameters reaches one conclusion. A candidate who tracks the gradient through the factorisation reaches the opposite, and correct, one. The gap between those two answers is exactly what the interviewer is probing.
This deep dive rebuilds the argument from the LoRA update equation, traces the gradient into each factor, explains the wide-model feature-learning view, and then turns to the practical recipe and the traps that surround it.
The LoRA factorisation and its initialisation
LoRA freezes the pretrained weight and learns a low-rank correction. The effective weight is the base plus a scaled product of two small matrices. B is tall, with shape d × r, and A is wide, with shape r × k, where r is the rank.
The initialisation is deliberate and asymmetric. A is filled with small Gaussian values. B is set to exactly zero. This means the product B·A is zero at step zero, so the adapter starts as a no-op and the model reproduces the base behaviour before any training.
That clean start is useful, but it plants the seed of the problem. One factor begins at zero and the other does not. The two factors are not interchangeable during the first phase of training, and a single shared learning rate quietly treats them as if they were.
It also matters that the two matrices have different shapes. B maps from the rank dimension up to the model dimension, while A maps from the input dimension down to the rank. They sit on opposite sides of the bottleneck, so the same gradient signal reaches each through a different path. Holding both to one rate ignores that they live in different parts of the computation, with different scales, from the very first step.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
| Aspect | Vanilla LoRA | LoRA+ |
|---|---|---|
| Learning rates | Single shared rate for A and B | Separate rates; B about 16 times A |
| Init asymmetry handled | No; B under-trains from zero start | Yes; higher B rate rebalances steps |
| Extra cost | None | Negligible; one config line |
| Main benefit | Baseline parameter efficiency | Faster convergence, small quality gain |
| Tuning burden | Tune one learning rate | Tune base rate plus the ratio |
Real products, models, and research that use this idea.
- Hugging Face PEFT exposes a LoRA+ learning-rate ratio option, letting practitioners split A and B rates with one config field on Llama 4 and Qwen fine-tunes.
- Unsloth's fine-tuning recipes document the LoRA+ ratio as a near-free convergence tweak when adapting open bases like Llama 3.1 8B.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy does swapping the A and B initialisation, putting the zero on A, not simply move the problem?
Trace which factor the output passes through and which gradient gets bottlenecked. The asymmetry is about which factor is zero at step zero and how its gradient scales, so reason about the per-factor effective step in either configuration.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Getting the ratio backwards by reasoning from parameter counts. The driver is the zero initialisation of B and its bottlenecked gradient, not how many parameters each factor holds.
60 second bullets to scan on the way to the call.
Why B is initialised at zero and A as Gaussian
How each factor's gradient depends on the other factor's magnitude
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.