A LoRA-wrapped projection has forward `h = x · (W_0 + (α/r) · B · A)`. During backprop, what gradient (if any) does each of `W_0`, `A`, and `B` receive? Write the shapes / chain-rule expressions for `dL/dA` and `dL/dB`, and explain why `W_0` is untouched.
Free · 2 AI evals / day
**LoRA** wraps a frozen linear layer `W_0` with a small trainable update made of two low-rank matrices `A` and `B`. Using the canonical Hu et al. convention (`A: r × d_in`, `B: d_out × r`), the forward is `h = (W_0 + (α/r) · B · A) · x`.
During backprop, the upstream gradient `dL/dh` enters the layer. It flows through both additive paths.
The frozen path through `W_0` does compute a gradient mathematically, but because `W_0.requires_grad=False`, the optimizer never applies an update. The weight stays exactly where pretraining left it.
The trainable path through `B · A` is where all the learning happens. Differentiating `(α/r) · B · A · x` with respect to `A` and `B` produces two gradients:
- `dL/dA = (α/r) · Bᵀ · (dL/dh) · xᵀ` (summed over the batch axis).
- `dL/dB = (α/r) · (dL/dh) · (A · x)ᵀ`.
Both carry the `(α/r)` scaling, and both route through the other factor. That is why `A` and `B` co-adapt: every step of `A` depends on the current value of `B`, and vice versa.
The rank-`r` bottleneck between them is what makes LoRA cheap: the gradient signal gets compressed to `r` dimensions before flowing out.
The forward pass `h = (W_0 + (α/r) · B · A) · x` expands additively to `h = W_0 · x + (α/r) · B · A · x`. Backprop treats these as two parallel paths. (Shape convention: `A: r × d_in`, `B: d_out × r`, x as column vector, per Hu et al. 2021.)
The **frozen path** through `W_0` is straightforward: `dL/dW_0 = (dL/dh) · xᵀ` is mathematically valid, but since `W_0.requires_grad=False`, the optimizer ignores it. No update reaches the base weights, which is the whole point of parameter-efficient adaptation.
The **trainable path** carries all the learning. Apply the chain rule to `(α/r) · B · A · x`:
- `dL/dA = (α/r) · Bᵀ · (dL/dh) · xᵀ`, shape `r × d_in`, matching `A`.
- `dL/dB = (α/r) · (dL/dh) · (A · x)ᵀ`, shape `d_out × r`, matching `B`.
Two conceptual takeaways. First, `A` and `B` **co-adapt**: each gradient routes through the current value of the other, so updates are not independent. Second, the `(α/r)` factor survives the differentiation, which is why naively doubling rank without rescaling α halves the effective update magnitude. That coupling is also why LoRA initializes `A` with Gaussian noise and `B` with zeros: the initial update is zero so training starts from the base model, then `B` ramps up and learns alongside `A`.
Quick reference
TL;DR
Only A and B receive optimizer updates; W_0 is frozen. dL/dA routes the upstream gradient through B-transpose times x, and dL/dB routes it through x·A-transpose, both scaled by alpha over rank.
Picture a giant frozen calculator (the base layer) with a tiny attachable adjustment dial bolted on the side. The dial itself is made of two small parts wired in series. When you crank back through the error signal, the big calculator does not move because someone glued its knobs in place. The two small dial parts do move, and they move in a coupled way: turning the first part changes how much the second part needs to turn, and vice versa. So they always end up adjusting together, never independently, and the tiny dial ends up steering the whole big calculator without ever changing it.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Domains requiring large representational drift from the base model, like a new modality or a very distant language The rank-r bottleneck on the gradient compresses adaptation to r dimensions; if the task needs movement far outside that subspace, full fine-tuning will beat LoRA.
Inference latency is the binding constraint and adapter swapping is not needed After training, merging the LoRA update back into W_0 gives identical cost to a full fine-tune at inference, so the LoRA wrapper adds nothing.
Very small base models where full fine-tuning fits comfortably in memory The memory and parameter-count savings of LoRA matter most at 7B and above; on a 350M model, full fine-tuning is simpler and usually trains faster per epoch.
Continual learning across many sequential tasks without explicit adapter management Repeatedly stacking LoRA updates on the same base risks interference across tasks; you need either per-task adapters or full fine-tuning with replay.
Quick scan · 2–4 min
Everything important, quickly.
5 min: forward decomposition + frozen vs trainable paths + dL/dA and dL/dB chain rule + (α/r) scaling + coupling between A and B + rank-r bottleneck + zero-init B + rsLoRA fix.
Side-by-side
Tensor
Receives gradient?
Optimizer step?
Why
W_0 (d_out × d_in)
Yes (mathematically)
No
requires_grad=False; PEFT often skips computation entirely
A (r × d_in)
Yes
Yes
dL/dA = (α/r) · Bᵀ · (dL/dh) · xᵀ
B (d_out × r)
Yes
Yes
dL/dB = (α/r) · (dL/dh) · (A · x)ᵀ
Real-world examples
Real products, models, and research that use this idea.
1Hugging Face PEFT is the canonical 2026 LoRA implementation; its `LoraLayer.forward` exactly matches the additive decomposition and its parameter-grouping skips W_0 in the optimizer.
2Unsloth optimizes LoRA backward by fusing the dL/dA and dL/dB kernels and skipping the dL/dW_0 buffer entirely, giving roughly 2x training speedup on Llama 4.
3rsLoRA (rank-stabilized LoRA) replaces α/r with α/√r so doubling rank does not silently halve the effective update magnitude; default in newer Axolotl and LLaMA-Factory recipes.
4DoRA splits the LoRA update into magnitude and direction components, then runs the same backward shape on each, an extension that ships in PEFT 0.10+.
5QLoRA combines LoRA backward with 4-bit quantized W_0, where the frozen base is dequantized on the forward and discarded; gradient computation through the dequantized W_0 is skipped to save memory.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy is B initialized to zero and A to small random values rather than both random?
Sign in or go Pro to see the approach for this follow-up.
QWhere does QLoRA save memory beyond standard LoRA, and what is the gradient story for the quantized W_0?
Sign in or go Pro to see the approach for this follow-up.
QHow does the (α/r) scaling interact with the learning rate, and why does rsLoRA prefer α/√r?
Sign in or go Pro to see the approach for this follow-up.
QCan you merge the LoRA update back into W_0 for inference, and what changes if you do?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Forgetting the alpha over rank scaling on the gradient and doubling rank without halving alpha, which silently halves the effective update magnitude even though the optimizer config looks unchanged.