**LoRA** trains tiny low-rank adapters on top of a frozen base model. You decide which weight matrices get an adapter, and that choice is the target-modules decision.
The classic minimal config patches only the query and value projections inside attention, written W_q and W_v. It is cheap, fast, and adds few trainable parameters. The all-linear config patches everything: query, key, value, output, plus the feed-forward gate, up, and down projections.
Across open-weight benchmarks, all-linear usually scores higher than W_q+W_v. The catch is cost. All-linear roughly doubles the trainable parameters and makes the saved adapter a bit larger. So the trade is quality versus parameter count.
The wrong answers in this question invent rules that do not exist. Sliding-window attention is just a masking pattern; it does not change which linear layers LoRA can touch. Mixture-of-experts routing still works fine with W_q+W_v. Strip those distractors and the real lever is clear: spend more parameters for more quality when you have the data to justify it.
The honest answer is a quality versus parameter-count trade. Targeting **all-linear** modules (Q, K, V, O plus the MLP gate, up, and down) consistently lifts task benchmarks over the W_q+W_v default. The price is roughly twice the trainable parameters and a slightly larger merged adapter.
So all-linear is the right default when quality matters more than parameter count, and you have enough data to fill those parameters. With a healthy dataset, the extra capacity captures task structure that two attention projections cannot.
The trap is assuming more modules are always better. With under a thousand examples, the larger adapter overfits. There you want fewer modules and a lower rank, not more. The small-data option in this question inverts the real rule.
Two distractors invent constraints. Sliding-window attention is a mask pattern, not a reason to add the output projection. Mixture-of-experts routing is unaffected by whether you patch Q and V. Knowing those are noise is half the question.
Quick reference
TL;DR
Go all-linear when you want maximum task quality and can spend roughly twice the trainable parameters. Small datasets and tight budgets favour W_q+W_v.
LoRA bolts tiny trainable patches onto a frozen model. You choose which layers get a patch. Patching only the query and value projections is cheap and gentle, like adjusting two knobs. Patching every linear layer (queries, keys, values, output, and the feed-forward block) is like adjusting every knob, which fits the task more tightly but costs about twice the trainable parameters. More knobs help when you have lots of data and want the best score. With only a handful of examples, too many knobs let the model memorise the examples instead of learning the pattern. So the real choice is quality versus parameter count, not some hidden rule about attention masks.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Fine-tuning on fewer than a thousand labelled examples All-linear adds capacity that overfits scarce data. Prefer W_q+W_v with a lower rank.
Tight VRAM or storage budget for many stored adapters All-linear roughly doubles trainable parameters and enlarges each saved adapter, which compounds across tasks.
Quick behavioural nudge like light tone or format adjustment A minimal W_q+W_v adapter often suffices, so the extra modules add cost without measurable benefit.
Rapid ablation sweeps where iteration speed dominates The smaller module set trains faster, letting you explore data and rank choices before committing to all-linear.
Quick scan · 2–4 min
Everything important, quickly.
4 min: the quality vs parameter-count trade + all-linear as the default + the small-data overfitting flip + why the windowing and MoE distractors are false + rank and module interaction.
Side-by-side
Concern
W_q + W_v
All-linear (Q, K, V, O + MLP)
Trainable parameters
Lower (baseline)
Roughly 2× higher
Benchmark quality
Good, slightly behind
Consistently higher on most tasks
Best dataset size
Small to medium
Medium to large
Overfitting risk on small data
Lower
Higher (too much capacity)
Adapter and merge footprint
Smaller
Slightly larger
Real-world examples
Real products, models, and research that use this idea.
1Hugging Face PEFT defaults many recipes to all-linear via target_modules='all-linear', reflecting the quality-first benchmark pattern.
2Unsloth's Llama 4 and Qwen 3 fine-tuning notebooks target Q, K, V, O plus the MLP gate, up, and down projections for best results.
3Axolotl configs expose lora_target_modules so teams flip between W_q+W_v and all-linear per dataset size.
4The original LoRA work on GPT-3 adapted only W_q and W_v to show strong results at minimal parameter cost, the historical baseline this question references.
5QLoRA popularised all-linear targeting on quantised bases, letting practitioners fine-tune large open-weight models like Llama 4 on a single GPU.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QHow do LoRA rank and target-module count interact in the trainable-parameter budget?
Sign in or go Pro to see the approach for this follow-up.
QWhy does all-linear tend to overfit on small datasets while W_q+W_v generalises better?
Sign in or go Pro to see the approach for this follow-up.
QIf a teammate insists sliding-window attention requires targeting the output projection, how do you correct them?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Believing more target modules always help. With scarce data, the extra parameters overfit; the right default flips toward fewer modules and lower rank.