What does LoRA's α/r ratio control, and why is α = 2r a common default?

Attempt it

In the LoRA update `W = W_0 + (α/r) · B · A`, what does the scalar `α/r` actually control? Why do many configs set α to a fixed multiple of r (e.g., α = 2r)?

Free · 2 AI evals / day

Quick reference

TL;DR

The α/r scalar is a fixed step-size multiplier on the LoRA update. Setting α proportional to r keeps that scale constant so you can sweep rank without retuning.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Imagine a volume knob wired to two adapter dials. The adapter learns the shape of a correction; the knob controls how loud that correction gets applied. The knob is α/r, and you set it by hand, not by training. Now suppose you widen the adapter to give it more room to learn. If the knob is fixed, widening it quietly turns the volume DOWN, so the correction gets softer even though the adapter is bigger. People then blame the bigger adapter for working worse, when really the volume just dropped. The fix is to turn the knob up in step with the width. Then the volume stays the same no matter how wide you go, and you can compare widths fairly.

Key concepts

LoRA adapters Alpha scaling Adapter rank Parameter-efficient fine-tuning Hyperparameter search

Quick scan · 2–4 min

Everything important, quickly.

5 min: what α/r multiplies + why it is fixed not learned + its role as a learning-rate multiplier + the rank-coupling trap when α is held fixed + how α = 2r restores rank-invariance + the rsLoRA caveat.

Side-by-side

Choice	What happens to α/r as r grows	Effect on rank sweeps
α fixed (e.g. α = 16 always)	Ratio shrinks, update weakens	Higher rank looks worse from smaller steps, not capacity
α = 2r (ratio constant)	Ratio stays at 2, scale stable	Differences reflect capacity; learning rate transfers
rsLoRA (divide by sqrt r)	Effective scale falls more slowly with r	Better high-rank behaviour when 1/r over-shrinks

Real-world examples

Real products, models, and research that use this idea.

Hugging Face PEFT exposes lora_alpha and r as separate config fields, and community recipes for tuning Llama 4 commonly set lora_alpha to twice r.
Axolotl and Unsloth fine-tuning templates for DeepSeek V4 and Llama 4 default to α = 2r so rank sweeps stay comparable without learning-rate retuning.
The rsLoRA option in PEFT switches the denominator from r to the square root of r, directly targeting the over-shrinking of high ranks discussed in interviews.
QLoRA recipes that fine-tune quantised open-weight bases keep α proportional to r so the same learning rate transfers across a rank grid search.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy does the LoRA update need a 1/r factor at all, and what does B starting at zero have to do with it?

Sign in or go Pro to see the approach for this follow-up.

QIf α/r and the base learning rate both scale the effective step, are they fully redundant?

Sign in or go Pro to see the approach for this follow-up.

QHow does rsLoRA's 1/sqrt(r) scaling change the story, and when would you reach for it?

Sign in or go Pro to see the approach for this follow-up.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.