**Five PEFT methods**, each defined by one specific advantage over the others.
LoRA is the general default. It adds small low-rank adapter matrices to attention projections and freezes the rest of the model. With a few hundred million trainable parameters on a 7B base, you get roughly 90% of full fine-tuning quality at a fraction of the cost. It is the right starting point for most fine-tuning work.
QLoRA combines LoRA with 4-bit quantization of the frozen base model. The base weights live in 4-bit NF4 format and the LoRA adapters stay in bf16. This cuts VRAM roughly fourfold compared to LoRA on the same model, which is what lets you fit a 70B base on a single 48GB GPU.
IA3 uses even fewer parameters than LoRA. It learns per-channel scaling vectors that multiply existing activations, no matrix factorisation, no rank to choose. It is ideal for very small SFT datasets where bigger adapters would overfit.
DoRA decomposes each weight update into a magnitude (one scalar per column) and a direction (the column normalised). LoRA tunes only the direction. The decomposition closes most of the gap between LoRA and full fine-tuning at low rank.
AdaLoRA does not fix one rank for every layer. It allocates the rank budget non-uniformly based on a sensitivity score derived from an SVD-style parameterisation, spending more rank where it matters most.
**Each of the five PEFT methods solves one specific problem.**
LoRA is the standard. Low-rank BA adapters added to attention projections (and sometimes MLP projections), with the base model frozen. Trainable parameters drop by two to three orders of magnitude compared to full fine-tuning, and quality typically reaches 90% or more of full FT on standard SFT tasks. It is the right default for most fine-tuning work, supported everywhere in the 2026 ecosystem.
QLoRA combines LoRA with 4-bit quantization of the frozen base, using the NF4 format and double quantization for the constants. The LoRA adapters stay in bf16 so gradients flow normally. The result is roughly fourfold VRAM reduction compared to LoRA on the same model, which makes a 70B base fit on a single 48GB GPU. The defining feature is the quantization of the frozen base, not a change to LoRA itself.
IA3 (Infused Adapter by Inhibiting and Amplifying Inner Activations) learns per-channel scaling vectors only, applied to keys, values, and feedforward outputs. No matrix factorisation, no rank hyperparameter, just one scalar per channel. Trainable parameter count is far smaller than LoRA, often by another order of magnitude, which makes it ideal for very small SFT datasets where bigger adapters overfit.
DoRA decomposes each weight column into a magnitude (one scalar per column) and a unit-norm direction. The magnitudes are learned directly and LoRA is applied only to the direction. The decomposition recovers the magnitude-direction correlation pattern of full fine-tuning, which closes most of the LoRA-to-full-FT gap, especially at low rank.
AdaLoRA reparameterises the update in SVD form (P times Lambda times Q), then prunes low-importance Lambda entries to reallocate a fixed global rank budget across layers. Layers that need more rank get more, layers that need less get less, instead of fixing one r for all.
Quick reference
TL;DR
LoRA is the general default, QLoRA shrinks VRAM with 4-bit quantization, IA3 uses the fewest parameters, DoRA decomposes magnitude and direction, AdaLoRA reallocates rank across layers.
Picture five different tool kits for adjusting a giant machine without rebuilding it. LoRA is the standard kit, small, fast, and good enough for most jobs. QLoRA is the lightweight travel version that lets you carry a much bigger machine in your van. IA3 is a minimal pocket kit with just a few dials. DoRA is a precision kit that adjusts the length and the angle of each part separately. AdaLoRA is a smart kit that watches which parts of the machine need attention and moves the screwdrivers where they help most. Each kit was built to solve one specific problem the others did not handle well.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Full fine-tuning is feasible and quality is the only thing that matters PEFT methods trade some quality for parameter efficiency, full FT remains the gold standard when VRAM and compute are not constraints.
The task needs continual learning across many sequential adapters with merge then continue cycles Some PEFT methods complicate adapter merging or interfere with each other when stacked, plain LoRA with disciplined merging is often safer than the more elaborate methods.
The framework or tooling does not have a tested integration for the method you want PEFT method bugs are subtle and hard to detect, going off the well-supported path costs more in debugging than the method's marginal advantage usually buys.
Quick scan · 2–4 min
Everything important, quickly.
4 min: name the binding constraint each method addresses + LoRA as the default + QLoRA for VRAM + IA3 for tiny data + DoRA for low-rank quality + AdaLoRA for per-layer budgeting + how to combine methods like QLoRA plus DoRA.
Real-world examples
Real products, models, and research that use this idea.
1Hugging Face PEFT ships all five methods as first-class configs in 2026, so swapping between them is usually a one-line change to a config object.
2QLoRA is the default for community fine-tunes of Llama 3.1 70B and DeepSeek V4 on single-GPU rigs, because the VRAM math is what makes those runs feasible.
3IA3 is used in research SFTs on tiny corpora (a few hundred examples) where LoRA at typical ranks overfits and full fine-tuning is overkill.
4DoRA is shipped behind a one-flag toggle in Axolotl and Unsloth in 2026, used by teams who want near-full-FT quality at LoRA cost without changing the rest of the recipe.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy does QLoRA pair specifically with NF4 rather than int4 or another 4-bit format?
Sign in or go Pro to see the approach for this follow-up.
QCan you combine multiple PEFT methods, for example QLoRA plus DoRA?
Sign in or go Pro to see the approach for this follow-up.
QWhen would you pick IA3 over LoRA even if VRAM is not a constraint?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Calling QLoRA just LoRA but better. The quantization piece is the defining feature, it shrinks VRAM by 4x while keeping LoRA quality, but it does not change the underlying adapter math.