When does targeting all-linear with LoRA pay off vs. just W_q+W_v?
Go all-linear when you want maximum task quality and can spend roughly twice the trainable parameters. Small datasets and tight budgets favour W_q+W_v.
LoRA bolts tiny trainable patches onto a frozen model. You choose which layers get a patch. Patching only the query and value projections is cheap and gentle, like adjusting two knobs. Patching every linear layer (queries, keys, values, output, and the feed-forward block) is like adjusting every knob, which fits the task more tightly but costs about twice the trainable parameters. More knobs help when you have lots of data and want the best score. With only a handful of examples, too many knobs let the model memorise the examples instead of learning the pattern. So the real choice is quality versus parameter count, not some hidden rule about attention masks.
Detailed answer & concept explanation~8 min readEverything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Everything important, quickly.
4 min: the quality vs parameter-count trade + all-linear as the default + the small-data overfitting flip + why the windowing and MoE distractors are false + rank and module interaction.
| Concern | W_q + W_v | All-linear (Q, K, V, O + MLP) |
|---|---|---|
| Trainable parameters | Lower (baseline) | Roughly 2× higher |
| Benchmark quality | Good, slightly behind | Consistently higher on most tasks |
| Best dataset size | Small to medium | Medium to large |
| Overfitting risk on small data | Lower | Higher (too much capacity) |
| Adapter and merge footprint | Smaller | Slightly larger |
Real products, models, and research that use this idea.
- Hugging Face PEFT defaults many recipes to all-linear via target_modules='all-linear', reflecting the quality-first benchmark pattern.
- Unsloth's Llama 4 and Qwen 3 fine-tuning notebooks target Q, K, V, O plus the MLP gate, up, and down projections for best results.
- Axolotl configs expose lora_target_modules so teams flip between W_q+W_v and all-linear per dataset size.
- The original LoRA work on GPT-3 adapted only W_q and W_v to show strong results at minimal parameter cost, the historical baseline this question references.
- QLoRA popularised all-linear targeting on quantised bases, letting practitioners fine-tune large open-weight models like Llama 4 on a single GPU.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow do LoRA rank and target-module count interact in the trainable-parameter budget?
QWhy does all-linear tend to overfit on small datasets while W_q+W_v generalises better?
Don't say thisRed flags and common mistakes that signal junior thinking. Click to expand.
Red flags and common mistakes that signal junior thinking. Click to expand.
Believing more target modules always help. With scarce data, the extra parameters overfit; the right default flips toward fewer modules and lower rank.
The night-before-the-interview bullets. Scan these on the way to the call.
Primary sources. Skim if you want the original framing.
Same topic, related formats. Practice these next.