Walk through what happens during backprop for a LoRA-wrapped linear layer
A LoRA-wrapped projection has forward `h = x · (W_0 + (α/r) · B · A)`. During backprop, what gradient (if any) does each of `W_0`, `A`, and `B` receive? Write the shapes / chain-rule expressions for `dL/dA` and `dL/dB`, and explain why `W_0` is untouched.
Only A and B receive optimizer updates; W_0 is frozen. dL/dA routes the upstream gradient through B-transpose times x, and dL/dB routes it through x·A-transpose, both scaled by alpha over rank.
Picture a giant frozen calculator (the base layer) with a tiny attachable adjustment dial bolted on the side. The dial itself is made of two small parts wired in series. When you crank back through the error signal, the big calculator does not move because someone glued its knobs in place. The two small dial parts do move, and they move in a coupled way: turning the first part changes how much the second part needs to turn, and vice versa. So they always end up adjusting together, never independently, and the tiny dial ends up steering the whole big calculator without ever changing it.
Detailed answer & concept explanation~8 min readEverything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Everything important, quickly.
5 min: forward decomposition + frozen vs trainable paths + dL/dA and dL/dB chain rule + (α/r) scaling + coupling between A and B + rank-r bottleneck + zero-init B + rsLoRA fix.
| Tensor | Receives gradient? | Optimizer step? | Why |
|---|---|---|---|
| W_0 (d_out × d_in) | Yes (mathematically) | No | requires_grad=False; PEFT often skips computation entirely |
| A (r × d_in) | Yes | Yes | dL/dA = (α/r) · Bᵀ · (dL/dh) · xᵀ |
| B (d_out × r) | Yes | Yes | dL/dB = (α/r) · (dL/dh) · (A · x)ᵀ |
Real products, models, and research that use this idea.
- Hugging Face PEFT is the canonical 2026 LoRA implementation; its `LoraLayer.forward` exactly matches the additive decomposition and its parameter-grouping skips W_0 in the optimizer.
- Unsloth optimizes LoRA backward by fusing the dL/dA and dL/dB kernels and skipping the dL/dW_0 buffer entirely, giving roughly 2x training speedup on Llama 4.
- rsLoRA (rank-stabilized LoRA) replaces α/r with α/√r so doubling rank does not silently halve the effective update magnitude; default in newer Axolotl and LLaMA-Factory recipes.
- DoRA splits the LoRA update into magnitude and direction components, then runs the same backward shape on each, an extension that ships in PEFT 0.10+.
- QLoRA combines LoRA backward with 4-bit quantized W_0, where the frozen base is dequantized on the forward and discarded; gradient computation through the dequantized W_0 is skipped to save memory.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy is B initialized to zero and A to small random values rather than both random?
QWhere does QLoRA save memory beyond standard LoRA, and what is the gradient story for the quantized W_0?
QHow does the (α/r) scaling interact with the learning rate, and why does rsLoRA prefer α/√r?
Don't say thisRed flags and common mistakes that signal junior thinking. Click to expand.
Red flags and common mistakes that signal junior thinking. Click to expand.
Forgetting the alpha over rank scaling on the gradient and doubling rank without halving alpha, which silently halves the effective update magnitude even though the optimizer config looks unchanged.
The night-before-the-interview bullets. Scan these on the way to the call.
Primary sources. Skim if you want the original framing.
Same topic, related formats. Practice these next.