Drag each answer to line up with its matching prompt
LoRA
Very small SFT datasets (a few hundred examples) where over parameterised adapters would overfit; learns only per channel scaling vectors.
QLoRA
Multi-tenant serving with thousands of tiny tasks where storage per task must be minimal and the model itself stays frozen.
IA3
Memory constrained setup: single consumer GPU or fitting a 70B model on a 48 GB card; trades a bit of quality for 4-bit base storage.
Prefix tuning
Aggressive domain shift with large data and serious compute budget where the rank ceiling of any PEFT method would hurt.
Full fine-tuning
Default choice for instruction tuning and most domain adaptation when you have enough VRAM to load the base in bf16.
Each PEFT method has one defining trade-off. Match the scenario's scarcest resource, VRAM, data size, or per-task storage, to the method whose mechanism solves exactly that.
Think of customising a rental car you must hand back unchanged. LoRA is a clip-on steering cover that subtly changes how it drives. QLoRA is the same cover, but you first deflate the seats so the whole car fits in a tiny garage. IA3 is just turning a few knobs up or down, perfect when you only get a short test drive and don't want to overdo it. Prefix tuning slips a sticky note onto the dashboard the car reads first, cheap to swap per renter. Full fine-tuning means rebuilding the engine, powerful, but only worth it with a big workshop and lots of time. You pick by what is scarce: space, data, or per-renter storage.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Match questions like this one are not testing whether you can recite five method names. They test whether you can read a scenario, find the single constraint that binds, and pick the method whose mechanism was designed for exactly that constraint. Get the mechanisms right and the mapping becomes almost mechanical.
The core insight is that parameter-efficient fine-tuning methods are not a quality ladder where one always beats the next. They sit at different points along three independent axes: how much capacity the trainable parameters have, how much GPU memory the setup needs, and how much storage each task costs at serving time. A method that wins on one axis often loses on another.
So the right mental model is a decision based on scarcity. Ask what is actually scarce in the scenario, then name the method that spends the least of that scarce thing while still solving the task. This deep dive walks each method's mechanism, then maps every scenario in the question to the constraint it really tests.
LoRA: the low-rank delta default
LoRA freezes the base model and learns a small additive update beside each large weight matrix. Instead of training the full matrix, it trains two thin matrices whose product reconstructs a low-rank delta. The update is added to the frozen weight at inference.
The effective weight is the frozen base plus a scaled low-rank product:
Here the rank r controls capacity. A rank of 8 to 32 captures most adaptation directions for instruction tuning and domain adaptation, which is why LoRA is the default whenever the base fits in bf16. A second advantage matters in production: the delta can be merged back into the base, so a deployed LoRA adds zero inference latency. That merge property is what makes LoRA the safe first choice unless another constraint forces your hand.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
| Method | Mechanism | Picks it when |
|---|---|---|
| LoRA | Low-rank delta beside frozen weights | Base fits in bf16; you want the quality default |
| QLoRA | LoRA delta over a 4-bit frozen base | VRAM is the bottleneck |
| IA3 | Per-channel activation scaling vectors | Tiny dataset; overfitting is the risk |
| Prefix tuning | Learned soft tokens per layer | Many tiny tasks; per-task storage must be small |
| Full fine-tuning | Update all weights | Aggressive domain shift with large data |
Real products, models, and research that use this idea.
- Hugging Face PEFT ships LoRA, QLoRA, IA3, and prefix tuning behind one config, so teams swap methods by changing a few lines.
- QLoRA's 4-bit recipe made fine-tuning a 65B model on a single 48 GB GPU routine, and it is the default for community Llama 4 fine-tunes.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy does QLoRA recover most of LoRA's quality despite a 4-bit frozen base?
Talk about the 4-bit format matching a normal weight distribution, double quantisation of the scales, and gradients flowing only through the bf16 LoRA delta, so quantisation error never accumulates in the trained parameters.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Picking a PEFT method by popularity rather than by the scarce resource. Each method optimises a different constraint, VRAM, data size, or per-task storage, so the right answer depends on which one binds.
60 second bullets to scan on the way to the call.
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.