LoRA vs Full Fine-tuning
Parameter-efficient adaptation vs updating all model weights
LoRA reaches 90-99% of full fine-tuning quality for a fraction of the compute and storage; use full fine-tuning only when that last margin is genuinely worth it.
LoRA (Low-Rank Adaptation)
Glossary →LoRA freezes the original model weights and injects small trainable low-rank matrices (rank 4-64) into attention layers. Typically modifies <1% of total parameters while achieving 90-99% of full fine-tuning quality.
Best for: Cheap, multi-tenant, hot-swappable adaptation.
Full Fine-tuning
Glossary →Full fine-tuning updates every parameter in the model on the target dataset. Maximum expressiveness but requires storing a complete copy of the model per task and significantly more GPU memory.
Best for: Maximum quality on one high-value task.
At a glance
| Dimension | LoRA (Low-Rank Adaptation) | Full Fine-tuning |
|---|---|---|
| Parameters trained | <1% (low-rank matrices only) | 100% (all weights) |
| GPU memory | Low (fits on consumer GPUs) | High (needs multi-GPU for large models) |
| Training time | 10-100x faster | Baseline |
| Storage per task | MBs (adapter weights only) | GBs (full model copy) |
| Quality vs base | 90-99% of full fine-tuning | 100% (ceiling) |
| Best for | Rapid experimentation, multi-tenant, budget-conscious | Maximum quality, single high-value deployment |
Key differences
- 1LoRA trains <1% of parameters; full fine-tuning trains 100%
- 2LoRA adapters are small (MBs) and can be hot-swapped; full fine-tuning produces a complete model copy (GBs)
- 3LoRA reduces catastrophic forgetting by keeping base weights frozen
- 4Full fine-tuning can achieve marginally better performance on domain specific tasks
- 5LoRA enables multi-tenant serving (one base model, many adapters); full fine-tuning needs separate deployments
In the interview
- Claiming LoRA and full fine-tuning are equivalent in every scenario
- Assuming full fine-tuning always wins on quality without a real eval
- Ignoring catastrophic forgetting risk in full fine-tuning
- Not knowing multi-tenant serving depends on LoRA-style adapters
How to choose
Default to LoRA. Reach for full fine-tuning only when the marginal quality genuinely earns the cost.
Common misconceptions
Myth: LoRA is only for hobbyists.
Reality: LoRA and QLoRA are the production default for adapting large models under budget and for serving many customers off one base.
Myth: Full fine-tuning always produces the best model.
Reality: Full fine-tuning risks catastrophic forgetting and often loses to a well-tuned LoRA on the actual eval, especially with small datasets.
Memory aid
Full fine-tuning is repainting the whole car; LoRA is snapping on a new dashboard. Both look great; one costs a lot less.
Can you combine them?
Yes. A common pattern is: full fine-tune on a large general domain corpus first, then apply LoRA adapters for specific tasks on top. QLoRA (quantized LoRA) further reduces memory by loading the base model in 4-bit.
Related topics
Related comparisons
DDP vs FSDP vs ZeRO
Three ways to shard distributed LLM training across GPUs
Distillation vs Quantization
Two orthogonal ways to shrink a large language model
DPO vs PPO
Direct Preference Optimization vs Proximal Policy Optimization for RLHF
In-context learning vs Fine-tuning
Adapt in the prompt, or adapt in the weights
RAG vs Fine-tuning
Two approaches to giving LLMs domain specific knowledge