In LoRA you add a small detour to a frozen weight: the update is `B·A`, where A is a wide matrix and B is a tall one. At the start of training, A is filled with small random numbers and **B** is set to exactly zero, so the detour starts as nothing and the model behaves like the original.
Here is the catch. To update B, the gradient has to pass through A, whose numbers are tiny. To update A, the gradient has to pass through B, which starts at zero. With a single shared learning rate, B ends up barely moving while A moves more. The two matrices learn at very different effective speeds.
**LoRA+** fixes this by giving B a much higher learning rate than A, around sixteen times higher. This balances how fast each matrix actually learns useful features.
The practical payoff: faster convergence and a small accuracy bump, for free, since you only change one number in your config. It matters most for harder tasks and wider models where the imbalance bites hardest.
**LoRA+** assigns separate learning rates to the two low-rank factors. B gets a much higher rate than A, with a ratio around sixteen.
The reason is the asymmetry baked into LoRA at initialisation. A starts as small Gaussian values and B starts at zero. The gradient into B passes through A, so it is scaled by A's tiny magnitude. The gradient into A passes through B, which is zero at the start. A single shared learning rate cannot make both factors move at a healthy, balanced pace.
The theory frames this through feature learning in the wide-model limit. A shared rate is provably suboptimal because the per-factor effective step sizes are mismatched. Lifting B's rate restores balanced learning and lets the adapter extract richer features.
The practical recipe is one tunable ratio. The split is:
$$\eta_B = \lambda \cdot \eta_A, \qquad \lambda \approx 16$$
The ratio is task dependent, so treat sixteen as a starting point and sweep it. The benefit is faster convergence and a modest quality gain at essentially zero extra cost, since you only split one hyperparameter.
Quick reference
TL;DR
B starts at zero and its gradient is bottlenecked, so a single shared learning rate under-trains it. LoRA+ gives B a roughly 16x higher rate than A.
Picture two people pushing a heavy cart together. One person is already leaning into it, warmed up and moving. The other is standing still with arms at their sides, starting from a dead stop. If you tell both to push with the same effort, the one starting cold barely helps, because they first have to overcome being frozen in place. The cart drifts unevenly and crawls along. So a smart coach tells the cold starter to push much harder than the warmed-up one, roughly sixteen times as hard, until both are shoving the cart at the same real pace. Now their combined effort moves the cart smoothly and quickly, instead of one person doing almost everything while the other lags behind the whole way. Match the effort to who is behind.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Tiny rank one adapters on a very small style task with plenty of training budget The convergence gain is marginal and a plain shared rate reaches the same quality.
A pipeline where the optimiser cannot assign separate parameter groups LoRA+ needs distinct rate groups for A and B; without that support you cannot apply the split.
Early instability where the loss spikes when B's rate is pushed up Larger B steps can destabilise training; lower the ratio or the base rate before continuing.
Full fine-tuning or methods without an explicit B and A factorisation There is no zero-initialised B factor to rebalance, so the LoRA+ rationale does not apply.
Quick scan · 2–4 min
Everything important, quickly.
5 min: LoRA factorisation recap, zero-init asymmetry, per-factor gradient bottleneck, the feature-learning argument, the ratio recipe and its tuning, then the production caveats and framework support.
Side-by-side
Aspect
Vanilla LoRA
LoRA+
Learning rates
Single shared rate for A and B
Separate rates; B about 16 times A
Init asymmetry handled
No; B under-trains from zero start
Yes; higher B rate rebalances steps
Extra cost
None
Negligible; one config line
Main benefit
Baseline parameter efficiency
Faster convergence, small quality gain
Tuning burden
Tune one learning rate
Tune base rate plus the ratio
Real-world examples
Real products, models, and research that use this idea.
1Hugging Face PEFT exposes a LoRA+ learning-rate ratio option, letting practitioners split A and B rates with one config field on Llama 4 and Qwen fine-tunes.
2Unsloth's fine-tuning recipes document the LoRA+ ratio as a near-free convergence tweak when adapting open bases like Llama 3.1 8B.
3Teams fine-tuning DeepSeek V4 and Mistral variants with the TRL plus PEFT stack sweep the LoRA+ ratio alongside rank and alpha during hyperparameter search.
4Axolotl configs ship a loraplus_lr_ratio setting so users enable the higher B rate without writing custom optimiser groups.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy does swapping the A and B initialisation, putting the zero on A, not simply move the problem?
Sign in or go Pro to see the approach for this follow-up.
QHow does the optimal LoRA+ ratio interact with the alpha over rank scaling and the chosen rank?
Sign in or go Pro to see the approach for this follow-up.
QWould LoRA+ still help if you used a different initialisation, such as both factors small but non-zero?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Getting the ratio backwards by reasoning from parameter counts. The driver is the zero initialisation of B and its bottlenecked gradient, not how many parameters each factor holds.