Same topic, related formats. Practice these next.
All-linear LoRA adapts every nn.Linear weight: the four attention projections plus the MLP gate, up, and down. It skips RMSNorm scale and the embedding matrix.
Picture a workshop full of machines. Most are the same kind: a mixing box that takes stuff in and pushes new stuff out. The all-everything rule clips a small, cheap dial onto every one of those mixing boxes, so you can re-tune the whole workshop without rebuilding any machine. But two things on the floor are not mixing boxes. One is a tiny volume knob that only makes a signal louder or softer. The other is a giant phone book that just looks up an answer when you give it a name. The rule is strict: clip dials only onto the real mixing boxes. Leave the little volume knob and the lookup book alone, because they do not mix anything, they only adjust or fetch.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Everything important, quickly.
5 min: define all-linear as the nn.Linear match rule, name the seven projections it hits, explain why RMSNorm and embeddings are skipped, then cover modules_to_save and the query and value only contrast.
| Module | PyTorch type | Hit by all-linear? |
|---|---|---|
| W_q, W_k, W_v, W_o | nn.Linear | Yes |
| MLP gate / up / down | nn.Linear | Yes |
| RMSNorm scale | Learned vector (Parameter) | No |
| Token embedding | nn.Embedding | No |
| Output LM head | nn.Linear (often tied) | No (excluded by guard) |
Real products, models, and research that use this idea.
What an interviewer would ask next. Try answering before peeking at the approach.
Red flags and common mistakes that signal junior thinking. Click to expand.
Assuming all-linear touches every trainable parameter. It targets only nn.Linear modules, so RMSNorm scale vectors and the embedding lookup are left frozen.
The night-before-the-interview bullets. Scan these on the way to the call.
Primary sources. Skim if you want the original framing.