All-linear LoRA adapts every nn.Linear weight: the four attention projections plus the MLP gate, up, and down. It skips RMSNorm scale and the embedding matrix.
Picture a workshop full of machines. Most are the same kind: a mixing box that takes stuff in and pushes new stuff out. The all-everything rule clips a small, cheap dial onto every one of those mixing boxes. Then you can re-tune the whole workshop without rebuilding any machine. But two things on the floor are not mixing boxes. One is a tiny volume knob that only makes a signal louder or softer. The other is a giant phone book that just looks up an answer when you give it a name. The rule is strict: clip dials only onto the real mixing boxes. Leave the little volume knob and the lookup book alone, because they do not mix anything. They only adjust or fetch.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
What all-linear covers comes down to how PEFT resolves a LoRA target spec. The string all-linear is not a vague intent like 'adapt the important layers'. It is a precise instruction with a precise implementation: walk the model's module tree and wrap every module whose type is nn.Linear.
The query, value, and output projections are matrix multiplies, so they are nn.Linear, so they are wrapped. The SwiGLU MLP projections are matrix multiplies too, so they are wrapped. The RMSNorm scale and the token embedding are not matrix multiplies, so they are skipped, even though both carry learnable weights.
The trap is the word 'layer'. Plenty of things in a transformer are layers in the loose sense, and several of them hold trainable parameters. But all-linear does not match on 'has a gradient' or 'is a layer'. It matches on the PyTorch type. That distinction between a parameter and an nn.Linear decides which modules get wrapped.
What all-linear means in PEFT
When you write target_modules='all-linear' in a LoraConfig, you are not naming layers by hand. You are handing PEFT a resolution rule. PEFT traverses the model's module tree and inspects the Python type of each submodule. It wraps every instance of nn.Linear (and its quantised variants, such as the 4-bit Linear that QLoRA uses) with a LoRA adapter.
The key word is type. The match is structural, not semantic. PEFT does not understand that a module is 'the value projection' or 'the gate'. It only sees that the module is an nn.Linear, and that is enough to wrap it. This is why all-linear generalises across architectures without you renaming targets for each new model family. The same spec works on Llama, Qwen, Mistral, and DeepSeek even though their layers carry different attribute names. It works because the resolver keys off the type rather than the name.
The alternative to all-linear is naming targets explicitly, for example a list like ['q_proj', 'v_proj']. That requires you to know the exact attribute names a given model uses, and those names differ across families. All-linear removes that fragility by matching on type. The trade-off is that you give up fine control: you cannot exclude one specific projection with the string alone. You have to fall back to an explicit list.
There is one important guard. PEFT excludes the final language-model head from the all-linear set. The head is large and, in some models, tied to the input embedding. You can still adapt it by naming it explicitly. So all-linear is precisely: every nn.Linear except the output head.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
| Module | PyTorch type | Hit by all-linear? |
|---|---|---|
| W_q, W_k, W_v, W_o | nn.Linear | Yes |
| MLP gate / up / down | nn.Linear | Yes |
| RMSNorm scale | Learned vector (Parameter) | No |
| Token embedding | nn.Embedding | No |
| Output LM head | nn.Linear (often tied) | No (excluded by guard) |
Real products, models, and research that use this idea.
- Hugging Face PEFT exposes target_modules='all-linear' as a one-line option that wraps every nn.Linear except the output layer. It works on Llama, Qwen, Mistral and other open models.
- Unsloth's example LoRA notebooks, such as those for Llama 3.1 8B, list all attention plus MLP projections, matching the all-linear set.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy does PEFT exclude the LM head from all-linear even though it is an nn.Linear?
PEFT excludes the output layer. Reason about the head's size (vocabulary by model width), weight tying with the embedding in some models, and how rarely a task needs the head itself adapted. Note that you can still target it by name.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Assuming all-linear touches every trainable parameter. It targets only nn.Linear modules, so RMSNorm scale vectors and the embedding lookup are left frozen.
60 second bullets to scan on the way to the call.
What all-linear resolves to in a SwiGLU transformer block
Why RMSNorm is not an nn.Linear despite having learned weights
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.