LoRA+: why does B want a higher learning rate than A?

In LoRA you add a small detour to a frozen weight: the update is `B·A`, where A is a wide matrix and B is a tall one. At the start of training, A is filled with small random numbers and **B** is set to exactly zero, so the detour starts as nothing and the model behaves like the original. Here is the catch. To update B, the gradient has to pass through A, whose numbers are tiny. To update A, the gradient has to pass through B, which starts at zero. With a single shared learning rate, B ends up barely moving while A moves more. The two matrices learn at very different effective speeds. **LoRA+** fixes this by giving B a much higher learning rate than A, around sixteen times higher. This balances how fast each matrix actually learns useful features. The practical payoff: faster convergence and a small accuracy bump, for free, since you only change one number in your config. It matters most for harder tasks and wider models where the imbalance bites hardest.

**LoRA+** assigns separate learning rates to the two low-rank factors. B gets a much higher rate than A, with a ratio around sixteen. The reason is the asymmetry baked into LoRA at initialisation. A starts as small Gaussian values and B starts at zero. The gradient into B passes through A, so it is scaled by A's tiny magnitude. The gradient into A passes through B, which is zero at the start. A single shared learning rate cannot make both factors move at a healthy, balanced pace. The theory frames this through feature learning in the wide-model limit. A shared rate is provably suboptimal because the per-factor effective step sizes are mismatched. Lifting B's rate restores balanced learning and lets the adapter extract richer features. The practical recipe is one tunable ratio. The split is: $$\eta_B = \lambda \cdot \eta_A, \qquad \lambda \approx 16$$ The ratio is task dependent, so treat sixteen as a starting point and sweep it. The benefit is faster convergence and a modest quality gain at essentially zero extra cost, since you only split one hyperparameter.