A LoRA wrapped projection has forward `h = x · (W_0 + (α/r) · B · A)`. During backprop, what gradient (if any) does each of `W_0`, `A`, and `B` receive? Write the shapes / chain rule expressions for `dL/dA` and `dL/dB`, and explain why `W_0` is untouched.
Only A and B receive optimizer updates; W_0 is frozen. dL/dA routes the upstream gradient through B-transpose times x, and dL/dB routes it through x·A-transpose, both scaled by alpha over rank.
Picture a giant frozen calculator (the base layer) with a tiny attachable adjustment dial bolted on the side. The dial itself is made of two small parts wired in series. When you crank back through the error signal, the big calculator does not move because someone glued its knobs in place. The two small dial parts do move, and they move in a coupled way: turning the first part changes how much the second part needs to turn, and vice versa. So they always end up adjusting together, never independently, and the tiny dial ends up steering the whole big calculator without ever changing it.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
LoRA, Low-Rank Adaptation, is the dominant parameter-efficient fine-tuning method in 2026. Its appeal is straightforward: rather than updating all of a 7B model's weights, you wrap each target linear layer with a small additive trainable correction made of two low-rank matrices, then train only those. The forward pass becomes h = x · (W_0 + (α/r) · B · A), where W_0 is the frozen base weight, A and B are the rank-r adapters, and (α/r) is a scaling factor.
The backward pass is where the interview signal lives. Anyone can quote the forward equation. Knowing what gradient flows where, and why the resulting update structure is the cheap but effective compromise it is, requires actually working through the chain rule.
The deep dive walks through three pieces: why W_0 receives no update despite sitting on the gradient graph, the exact chain-rule expressions for dL/dA and dL/dB, and the operational consequences of the (α/r) scaling and the rank-r bottleneck.
Decomposing the forward into two paths
Start by expanding the forward additively: h = W_0 · x + (α/r) · B · A · x. The first term is the unchanged base layer. The second is the LoRA adapter contribution, structured as a sequence of two small matmuls separated by the rank-r intermediate dimension. (Hu et al. (2021) shape convention: A: r × d_in is the down-projection, B: d_out × r is the up-projection, with x as a column vector.)
Because the two terms add, backprop treats them as parallel paths. The upstream gradient dL/dh enters the layer and is distributed to both paths according to the sum rule of differentiation. There is no interference between them at this level; each path computes its own contribution to the gradients of its inputs.
This additive structure is what makes LoRA an adapter rather than a replacement. The base model is mathematically still there, doing what it always did. The adapter rides alongside, adding a small low-rank correction. That correction is the only thing the optimizer ever changes.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
| Tensor | Receives gradient? | Optimizer step? | Why |
|---|---|---|---|
| W_0 (d_out × d_in) | Yes (mathematically) | No | requires_grad=False; PEFT often skips computation entirely |
| A (r × d_in) | Yes | Yes | dL/dA = (α/r) · Bᵀ · (dL/dh) · xᵀ |
| B (d_out × r) | Yes | Yes | dL/dB = (α/r) · (dL/dh) · (A · x)ᵀ |
Real products, models, and research that use this idea.
- Hugging Face PEFT is the canonical 2026 LoRA implementation; its `LoraLayer.forward` exactly matches the additive decomposition and its parameter-grouping skips W_0 in the optimizer.
- Unsloth optimizes LoRA backward by fusing the dL/dA and dL/dB kernels and skipping the dL/dW_0 buffer entirely, giving roughly 2x training speedup on Llama 4.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy is B initialized to zero and A to small random values rather than both random?
The initial LoRA contribution must be zero so training starts from exactly the base model. Zero-init B guarantees that; A being zero would also block all gradient flow into B because dL/dB routes through A · x.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Forgetting the alpha over rank scaling on the gradient and doubling rank without halving alpha, which silently halves the effective update magnitude even though the optimizer config looks unchanged.
60 second bullets to scan on the way to the call.
The additive decomposition of the LoRA forward into frozen and trainable paths
Why W_0 sees no optimizer update despite being on the gradient graph
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.