**LoRA** freezes the base model and learns two small matrices, B and A, whose product is added to a weight. The rank r is the shared inner dimension of those two matrices.
When you grow r from 16 to 64, both matrices get taller or wider in their r dimension. Their parameter count is proportional to r, so quadrupling r roughly quadruples how many numbers you are training.
There is a second, sneakier effect. LoRA does not add the raw product BA. It multiplies it by a scalar equal to alpha divided by r. If you raise r but leave alpha alone, that scalar shrinks by the same factor, so every update is scaled down to a quarter of its old size.
So two things move at once: more parameters, and smaller per-step updates. Training can look like it slowed down even though you added capacity.
The fix is the convention of tying alpha to r, for example alpha equal to twice r. Then the scaling factor stays roughly constant and only the capacity changes.
Raising rank touches two independent dials, and the question rewards keeping them apart.
The first dial is **parameter count**. The matrix B has shape d by r and A has shape r by k, so their combined size grows linearly with r. Going from 16 to 64 roughly quadruples the trainable parameters.
The second dial is the update magnitude. LoRA scales the learned product by the factor alpha over r. The forward path is:
$$h = W_0 x + \tfrac{\alpha}{r}\, BA\, x$$
That scalar lives in the forward pass, not the parameter list. So when r quadruples with alpha fixed, the factor shrinks fourfold. Every effective update is quartered, and training appears to slow.
The two effects combine, which is the whole point. You get more capacity but weaker steps. That is why practitioners tie alpha to r, commonly alpha equal to twice r. Holding the ratio constant means rank changes capacity without secretly rescaling the learning signal.
Quick reference
TL;DR
Raising LoRA rank 16 to 64 with α fixed quadruples trainable params but quarters the effective update, because the scaling factor is α divided by r.
Think of LoRA as a small dimmer switch bolted onto a big frozen lamp. The rank is how many tiny wires the dimmer has: more wires means it can shape light in more ways, so going from 16 to 64 wires roughly quadruples the parts you adjust. But there is also one master volume knob set to alpha divided by rank. When you add wires without turning alpha up, that master knob automatically turns down by the same factor. So you have built a richer dimmer, yet you have also dialled its overall brightness down to a quarter. To keep the brightness steady, you raise alpha alongside the rank.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Sweeping rank as a clean capacity ablation while leaving alpha pinned A fixed alpha turns the rank sweep into a hidden learning-rate sweep, so the ablation confounds capacity with step size.
Pushing rank very high while keeping the linear alpha over r form Gradient noise grows with rank, so the linear form can under-scale; rank-stabilised LoRA divides by the square root of rank instead.
Reusing an alpha tuned for a small rank on a much larger rank The effective update collapses and training stalls, which looks like the larger rank failing to help.
Treating alpha as a knob unrelated to the learning rate The scaling factor and learning rate both multiply the update, so changing one without checking the other misstates the true step size.
Quick scan · 2–4 min
Everything important, quickly.
4 min: separate the two dials, derive the param-count scaling, derive the alpha over r scaling, explain why they oppose, then cover the alpha tied to rank convention and rank-stabilised LoRA.
Side-by-side
Quantity
Depends on
Effect of r: 16 to 64 with alpha fixed
Trainable parameters
Shape of B and A
Roughly quadruples (grows linearly with r)
Scaling factor alpha over r
Forward-pass scalar
Shrinks fourfold
Effective update magnitude
Scaling factor times BA
Quartered, so training appears to slow
Fix
Tie alpha to r
Ratio held constant, only capacity changes
Real-world examples
Real products, models, and research that use this idea.
1Hugging Face PEFT exposes both lora_alpha and r, and its docs warn users to scale alpha with rank rather than leave it pinned.
2The rank-stabilised LoRA option (use_rslora) in PEFT divides by the square root of r so high-rank adapters of Llama 4 stay stable during fine-tuning.
3Unsloth and Axolotl recipes for fine-tuning Llama 4 and DeepSeek V4 default to alpha equal to twice the rank for exactly this decoupling reason.
4QLoRA fine-tunes of Mistral and Llama 4 commonly use rank 64 with alpha 16, an aggressive low-scale choice teams pick deliberately, not by accident.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy does the LoRA paper scale by alpha over r instead of just learning the magnitude inside BA?
Sign in or go Pro to see the approach for this follow-up.
QHow does rank-stabilised LoRA change the scaling, and why might dividing by the square root of r be better at high rank?
Sign in or go Pro to see the approach for this follow-up.
QIf you double the learning rate instead of doubling alpha, do you get the same training dynamics?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Assuming rank only affects parameter count. It also moves the α over r scaling factor, so raising rank with α fixed silently shrinks every update.