Match PEFT method to the scenario it fits best

PEFT stands for parameter-efficient fine-tuning. The idea is simple: instead of updating all of a model's billions of weights, you freeze the base and train a tiny number of new parameters. That keeps memory low and lets you store one small file per task. The trick is that each method makes a different bet. **LoRA** injects a small low-rank update next to the big weight matrices, so it learns a compact delta. It is the safe default when your GPU can hold the base model. QLoRA does the same thing but first squeezes the frozen base into 4-bit storage, so you can train a huge model on one card. You trade a sliver of quality for a big memory saving. IA3 is even lighter: it just learns a few scaling numbers that multiply activations. That tiny footprint resists overfitting when you only have a few hundred examples. Prefix tuning prepends learned virtual tokens instead, keeping the base frozen. Full fine-tuning updates everything, which you reach for only with lots of data and compute.

Every PEFT method optimises one scarce resource, so you match the scenario's binding constraint to the method's mechanism. **LoRA** is the default. It learns a low-rank delta beside each frozen weight matrix, giving strong quality whenever the base fits in bf16. QLoRA layers 4-bit quantisation of that frozen base underneath, so it wins when VRAM is the bottleneck, like fitting a 70B model on a 48 GB card. The mechanism is identical to LoRA; only the base storage changes. IA3 learns only per-channel scaling vectors that rescale activations. That tiny capacity is the point: with a few hundred examples, richer adapters overfit, so IA3's low parameter count is a feature. Prefix tuning prepends learned virtual tokens to every layer, keeping the base frozen and per-task storage minimal, which fits multi-tenant serving of thousands of tiny tasks. Full fine-tuning updates all weights. You only need it for aggressive domain shift with large data, where the low-rank ceiling of any adapter would cap quality. Two questions decide it fast. Is the bottleneck VRAM or quality? Is the dataset large enough that overfitting is unlikely?