Which of these are linear layers that 'all-linear' LoRA targets in a SwiGLU transformer?

Multi-select·Medium·4.0 · 0·~1 min·Relevant atCohere Databricks Meta Microsoft

Attempt it

Quick reference

TL;DR

All-linear LoRA adapts every nn.Linear weight: the four attention projections plus the MLP gate, up, and down. It skips RMSNorm scale and the embedding matrix.

Memory aid

Easy to grasp

Picture a workshop full of machines. Most are the same kind: a mixing box that takes stuff in and pushes new stuff out. The all-everything rule clips a small, cheap dial onto every one of those mixing boxes, so you can re-tune the whole workshop without rebuilding any machine. But two things on the floor are not mixing boxes. One is a tiny volume knob that only makes a signal louder or softer. The other is a giant phone book that just looks up an answer when you give it a name. The rule is strict: clip dials only onto the real mixing boxes. Leave the little volume knob and the lookup book alone, because they do not mix anything, they only adjust or fetch.

Key concepts

LoRA adapters Transformer block layers PEFT target modules SwiGLU MLP

Detailed answer & concept explanation

~7 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Quick scan · 2–4 min

Everything important, quickly.

5 min: define all-linear as the nn.Linear match rule, name the seven projections it hits, explain why RMSNorm and embeddings are skipped, then cover modules_to_save and the query and value only contrast.

Side-by-side

Module	PyTorch type	Hit by all-linear?
W_q, W_k, W_v, W_o	nn.Linear	Yes
MLP gate / up / down	nn.Linear	Yes
RMSNorm scale	Learned vector (Parameter)	No
Token embedding	nn.Embedding	No
Output LM head	nn.Linear (often tied)	No (excluded by guard)

Real-world examples

Real products, models, and research that use this idea.

Hugging Face PEFT exposes target_modules='all-linear' as a one-line default that wraps every nn.Linear when fine-tuning Llama 4 or Qwen 3.
Unsloth's LoRA recipes for Llama 4 and Gemma 3 default to all attention plus MLP projections, matching the all-linear set for maximum quality.
Axolotl config files for fine-tuning DeepSeek V4 commonly set lora_target_linear: true, which expands to the same all-linear resolution.
QLoRA training of a 70B base on a single A100 relies on all-linear targeting so the 4-bit frozen weights still receive adapters on every projection.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy does PEFT exclude the LM head from all-linear even though it is an nn.Linear?

QWhen would adapting only W_q and W_v beat adapting all-linear?

QHow do you adapt RMSNorm and embeddings if all-linear ignores them?

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Assuming all-linear touches every trainable parameter. It targets only nn.Linear modules, so RMSNorm scale vectors and the embedding lookup are left frozen.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

References

Primary sources. Skim if you want the original framing.