PEFT: unpack the acronym and the core problem it tackles.
PEFT freezes the base model and trains a tiny extra parameter set (often under 1% of weights) so one base can host many cheap task adapters.
Imagine a giant cookbook the chef has already memorised. Teaching that chef a brand-new cuisine the old way means rewriting every page, expensive, slow, and the cookbook gets enormous. PEFT does something cleverer. It leaves the cookbook untouched and clips a slim recipe card to a few key pages. The card is small, fast to write, and easy to swap. One chef can carry dozens of cards, one per cuisine, and pull the right one when needed. The cookbook (the frozen base model) never changes; the recipe cards (the tiny trainable adapters) carry the new behaviour. You get a per-task speciality without copying the whole book each time.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
PEFT, Parameter-Efficient Fine-Tuning, is one of the most consequential ideas of the post-2021 LLM era, and it is also one of the most commonly misunderstood. Candidates know the acronym. They can usually name LoRA. What separates a strong answer from a slogan is being able to explain why the family exists at all, what each method actually changes about the forward pass, and where the approach breaks down.
The headline motivation is operational, not theoretical. As base model sizes climbed from a few hundred million parameters into the tens and hundreds of billions, full fine-tuning stopped being a thing most teams could afford. A single 70B fine-tune needs roughly the model size in weights plus twice that in Adam optimizer state, putting the run on a multi-node cluster. The resulting checkpoint is another full copy of the model, 140 GB or more, and if you need a per-customer, per-language, or per-domain variant you multiply that storage cost by however many variants you ship.
PEFT solves the problem by changing what you train. Instead of updating every parameter, you freeze the entire base and train a tiny extra slice, typically well under one percent of the total weight count. The slice is small enough to fit on consumer hardware, small enough to store cheaply per task, and small enough to swap at request time. The frozen base is shared across every adapter you ship.
This deep dive walks through the three families of PEFT methods, explains where each one inserts capacity into the forward pass, covers the serving story that makes the approach economically attractive, and names the boundary cases where full fine-tuning is still the right choice.
Why full fine-tuning gets impractical fast
The optimizer-state problem
Full fine-tuning needs three things in GPU memory: the weights, the gradients, and the optimizer state. For Adam, the optimizer state is two extra tensors per parameter (first and second moment estimates), each the same size as the weights. In bf16 training that is roughly 6 bytes per parameter total, so a 70B model wants 420 GB just for the training state, before activations or gradient checkpoints.
The checkpoint problem
Assume you survive training. The artifact you ship is a full copy of the model, around 140 GB at bf16. Now multiply by every customer, language, domain, or experiment you want to maintain. A team running 50 fine-tunes pays 7 TB of storage and has to load a fresh model into memory every time the served variant changes.
Why this killed the old playbook
The pre-LLM transfer-learning recipe (fine-tune all weights of BERT for every task) was viable when BERT was 110M parameters. At 70B it is not. PEFT is the reaction to that scaling wall, and the entire post-2021 fine-tuning ecosystem assumes some form of PEFT by default.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
| Aspect | Full fine-tuning | PEFT (LoRA-style) |
|---|---|---|
| Trainable parameters | 100% of weights | Often under 1% |
| Optimizer memory | Roughly 3x model size | Roughly 3x the tiny delta |
| Checkpoint size per task | Full model copy | A few megabytes |
| Multi-task serving | One model per task in memory | One base hosts many adapters |
| Quality ceiling | Highest | Slightly lower, often within 1-2 points |
Real products, models, and research that use this idea.
- Hugging Face PEFT library ships LoRA, AdaLoRA, IA3, prefix tuning, and prompt tuning under one API used across the open-weight ecosystem.
- QLoRA fine-tunes of Llama 4 Maverick and DeepSeek V4 routinely run on a single 80 GB GPU thanks to 4-bit base quantization plus low-rank adapters.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy does PEFT typically not teach new factual knowledge well?
Knowledge lives broadly across the frozen base weights. A low-rank or small additive delta cannot rewrite that distribution; it can only shift behaviour at a narrow attention angle. Retrieval-augmented generation or continued pretraining is the right tool for fresh facts.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Treating PEFT as a single algorithm. PEFT is an umbrella covering LoRA, adapters, prefix tuning, prompt tuning, and BitFit, each with different placement, memory, and quality trade-offs.
60 second bullets to scan on the way to the call.
What the PEFT acronym expands to
Why full fine-tuning becomes impractical at 70B scale
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.