All-linear LoRA adapts every nn.Linear weight: the four attention projections plus the MLP gate, up, and down. It skips RMSNorm scale and the embedding matrix.
Picture a workshop full of machines. Most are the same kind: a mixing box that takes stuff in and pushes new stuff out. The all-everything rule clips a small, cheap dial onto every one of those mixing boxes, so you can re-tune the whole workshop without rebuilding any machine. But two things on the floor are not mixing boxes. One is a tiny volume knob that only makes a signal louder or softer. The other is a giant phone book that just looks up an answer when you give it a name. The rule is strict: clip dials only onto the real mixing boxes. Leave the little volume knob and the lookup book alone, because they do not mix anything, they only adjust or fetch.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
This question looks like architecture trivia, but it is really testing whether you understand how PEFT resolves a LoRA target spec. The string all-linear is not a vague intent like 'adapt the important layers'. It is a precise instruction with a precise implementation: walk the model's module tree and wrap every module whose type is nn.Linear.
Once you hold that definition in your head, every option sorts itself. The query, value, and output projections are matrix multiplies, so they are nn.Linear, so they are wrapped. The SwiGLU MLP projections are matrix multiplies too, so they are wrapped. The RMSNorm scale and the token embedding are not matrix multiplies, so they are skipped, even though both carry learnable weights.
The trap the question sets is the word 'layer'. Plenty of things in a transformer are layers in the loose sense, and several of them hold trainable parameters. But all-linear does not match on 'has a gradient' or 'is a layer'. It matches on the PyTorch type. That distinction between a parameter and an nn.Linear is the whole answer.
This deep dive defines all-linear precisely, names the seven projections in a SwiGLU block, explains why the two distractors fail the type test, and covers the escape hatch you use when you genuinely need to adapt norms or embeddings.
What all-linear actually means in PEFT
When you write target_modules='all-linear' in a LoraConfig, you are not naming layers by hand. You are handing PEFT a resolution rule. PEFT traverses the model's module tree, inspects the Python type of each submodule, and wraps every instance of nn.Linear (and its quantised variants, such as the 4-bit Linear that QLoRA uses) with a LoRA adapter.
The key word is type. The match is structural, not semantic. PEFT does not understand that a module is 'the value projection' or 'the gate'. It only sees that the module is an nn.Linear, and that is enough to wrap it. This is why all-linear generalises across architectures without you renaming targets for each new model family. The same spec works on Llama, Qwen, Mistral, and DeepSeek even though their layers carry different attribute names, because the resolver keys off the type rather than the name.
The alternative to all-linear is naming targets explicitly, for example a list like ['q_proj', 'v_proj']. That requires you to know the exact attribute names a given model uses, and those names differ across families. All-linear removes that fragility by matching on type. The trade-off is that you give up fine control: you cannot exclude one specific projection with the string alone, you have to fall back to an explicit list.
There is one important guard. PEFT excludes the final language-model head from the all-linear set, because that head is frequently tied to the input embedding. Adapting a tied projection that also feeds the output softmax causes problems. So all-linear is precisely: every nn.Linear except the output head.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
| Module | PyTorch type | Hit by all-linear? |
|---|---|---|
| W_q, W_k, W_v, W_o | nn.Linear | Yes |
| MLP gate / up / down | nn.Linear | Yes |
| RMSNorm scale | Learned vector (Parameter) | No |
| Token embedding | nn.Embedding | No |
| Output LM head | nn.Linear (often tied) | No (excluded by guard) |
Real products, models, and research that use this idea.
- Hugging Face PEFT exposes target_modules='all-linear' as a one-line default that wraps every nn.Linear when fine-tuning Llama 4 or Qwen 3.
- Unsloth's LoRA recipes for Llama 4 and Gemma 3 default to all attention plus MLP projections, matching the all-linear set for maximum quality.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy does PEFT exclude the LM head from all-linear even though it is an nn.Linear?
Think about weight tying between the embedding and the output head, and what happens to the loss surface if you low-rank adapt a tied projection that also feeds the softmax. Discuss the guard PEFT applies.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Assuming all-linear touches every trainable parameter. It targets only nn.Linear modules, so RMSNorm scale vectors and the embedding lookup are left frozen.
60 second bullets to scan on the way to the call.
What all-linear resolves to in a SwiGLU transformer block
Why RMSNorm is not an nn.Linear despite having learned weights
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.