What does LoRA's α/r ratio control, and why is α = 2r a common default?
In the LoRA update `W = W_0 + (α/r) · B · A`, what does the scalar `α/r` actually control? Why do many configs set α to a fixed multiple of r (e.g., α = 2r)?
The α/r scalar is a fixed step-size multiplier on the LoRA update. Setting α proportional to r keeps that scale constant so you can sweep rank without retuning.
Imagine a volume knob wired to two adapter dials. The adapter learns the shape of a correction; the knob controls how loud that correction gets applied. The knob is α/r, and you set it by hand, not by training. Now suppose you widen the adapter to give it more room to learn. If the knob is fixed, widening it quietly turns the volume DOWN, so the correction gets softer even though the adapter is bigger. People then blame the bigger adapter for working worse, when really the volume just dropped. The fix is to turn the knob up in step with the width. Then the volume stays the same no matter how wide you go, and you can compare widths fairly.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
LoRA fine-tunes a large model by freezing its pretrained weights and learning a small low-rank correction alongside them. The correction is the product of two thin matrices, and it gets added back into the frozen weight at every forward pass. The question is about the single scalar sitting in front of that correction: α divided by r.
This scalar looks innocent, but it is one of the most misunderstood numbers in parameter-efficient fine-tuning. People assume it is learned. It is not. People assume a bigger rank is always better. It is not, unless you hold this scalar fixed. And people sweep rank to find the best capacity, then draw exactly the wrong conclusion because the scalar quietly moved underneath them.
The goal of this deep dive is to make the scalar concrete. We will pin down what it multiplies, why it behaves like a learning rate, what goes wrong when you change rank without thinking about it, why the α = 2r convention fixes that, and where the convention itself breaks down.
What the scalar multiplies
Start from the update itself. The frozen base weight stays untouched, and the learned part is a low-rank product scaled by a fixed factor.
Here B A is the learned correction. The matrix A maps the input down to a small rank-r space, and B maps it back up to the full output dimension. Neither matrix is large: that is the whole point of the method, since you train far fewer parameters than the full weight. For a weight of size d-by-k, full fine-tuning touches d times k parameters, while LoRA touches only r times (d plus k). At a rank of 16 on a large projection, that is often a hundredfold reduction in trainable parameters.
The factor α/r multiplies that entire product. It does not change the direction of the correction, only its magnitude. So whatever shape BA learns, this scalar decides how loudly that shape is applied on top of the frozen weight. It is a global gain on the adapter delta, set once at config time.
It helps to separate two distinct quantities. The rank r controls capacity: how many independent directions the correction can span, and therefore how rich a behaviour change the adapter can express. The scalar α/r controls amplitude: how strongly that correction is pushed into the base weight per step. Capacity and amplitude are different axes, and the rest of this explanation is really about not confusing the two. The single most common mistake in practice is changing one of them and accidentally moving the other.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
| Choice | What happens to α/r as r grows | Effect on rank sweeps |
|---|---|---|
| α fixed (e.g. α = 16 always) | Ratio shrinks, update weakens | Higher rank looks worse from smaller steps, not capacity |
| α = 2r (ratio constant) | Ratio stays at 2, scale stable | Differences reflect capacity; learning rate transfers |
| rsLoRA (divide by sqrt r) | Effective scale falls more slowly with r | Better high-rank behaviour when 1/r over-shrinks |
Real products, models, and research that use this idea.
- Hugging Face PEFT exposes lora_alpha and r as separate config fields, and community recipes for tuning Llama 4 commonly set lora_alpha to twice r.
- Axolotl and Unsloth fine-tuning templates for DeepSeek V4 and Llama 4 default to α = 2r so rank sweeps stay comparable without learning-rate retuning.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy does the LoRA update need a 1/r factor at all, and what does B starting at zero have to do with it?
Trace the magnitude of the BA product as rank grows: more rank-one terms accumulate, so the typical norm scales with r. Dividing by r normalises that, and zero-init on B means the adapter starts contributing nothing and grows from there.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Treating α as a learned weight, or sweeping rank with α fixed. The second silently rescales the update so a larger rank looks worse when only the step size shrank.
60 second bullets to scan on the way to the call.
What the α/r scalar multiplies in the update
Why α/r is fixed rather than learned
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.