Why does B want a higher learning rate than A in LoRA+?
A's update acts on the full-width input, so its effect grows with width; B's acts on only r numbers. LoRA+ gives B a higher rate than A by a tuned ratio well above 1.
Picture two workers adjusting a big sound system. The first worker turns knobs on a wide mixing desk with thousands of input channels. Each small turn adds up across every channel, so even a tiny move changes the sound a lot. The second worker adjusts a small box with only a handful of dials. Each turn there changes the sound only a little. If you tell both to turn their knobs by the same amount, one of two things goes wrong. Either the first worker blasts the sound, or the second worker barely changes anything. So a smart engineer tells the second worker to turn much harder than the first, until both change the sound at a similar pace. The wider the mixing desk, the bigger that gap needs to be. Match the push to how much each move counts.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
LoRA+ is a one-line change to how you set learning rates in LoRA, and it routinely buys faster convergence plus a small accuracy bump. The idea sounds almost too small to matter. Instead of using one learning rate for the whole adapter, you give the B matrix a higher rate than the A matrix, by a ratio you tune. That asymmetry is the right call, and its direction is easy to get backwards.
The reason lies in how each factor's update reaches the output. A multiplies the full-width input, while B multiplies a vector of only r numbers. Once you see that, the gap in how far each update moves the output falls out. The corrective learning rate ratio then follows directly. The factorised structure that makes LoRA cheap is the same structure that makes a single learning rate the wrong tool.
The surface reasoning and the correct reasoning point in opposite directions. Counting parameters leads to one conclusion. Tracking the gradient through the factorisation leads to the opposite, and correct, one.
The LoRA factorisation and its initialisation
LoRA freezes the pretrained weight and learns a low-rank correction. The effective weight is the base plus a scaled product of two small matrices. B is tall, with shape d × r, and A is wide, with shape r × k, where r is the rank.
The initialisation is deliberate and asymmetric. A is filled with small Gaussian values. B is set to exactly zero. This means the product B·A is zero at step zero. So the adapter starts as a no-op and the model reproduces the base behaviour before any training.
That clean start is useful, but it is not what LoRA+ fixes. The asymmetry that matters comes from the shapes of the two factors.
The two matrices have different shapes. B maps from the rank dimension up to the model dimension, while A maps from the input dimension down to the rank. They sit on opposite sides of the bottleneck, so the same gradient signal reaches each through a different path. Holding both to one rate ignores that they live in different parts of the computation, with different scales, from the very first step.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
| Aspect | Vanilla LoRA | LoRA+ |
|---|---|---|
| Learning rates | Single shared rate for A and B | Separate rates; B higher than A by a tuned ratio |
| Width scaling handled | No; B learns too slowly | Yes; higher B rate rebalances steps |
| Extra cost | None | Negligible; one config line |
| Main benefit | Baseline parameter efficiency | Faster convergence, small quality gain |
| Tuning burden | Tune one learning rate | Tune base rate plus the ratio |
Real products, models, and research that use this idea.
- Hugging Face PEFT provides create_loraplus_optimizer, which takes a loraplus_lr_ratio (16 in its usage example) and builds separate optimizer groups for the A and B matrices.
- LoRA+ is reported to raise fine-tuning speed by up to 2x and performance by 1-2%, at almost no extra cost.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy does swapping the A and B initialisation, putting the zero on A, not simply move the problem?
Trace what each update multiplies. A's update still acts on the full-width input and B's on the r-dimensional Ax, whichever factor starts at zero. The paper's analysis covers both inits and still needs η_B much larger than η_A.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Getting the ratio backwards by reasoning from parameter counts, or blaming B's zero init. The driver is width: A's update is summed over the full-width input, B's over only r numbers.
60 second bullets to scan on the way to the call.
Why the effect of A's update grows with width and B's does not
Why the result holds whether B or A starts at zero
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.