LoRA vs full FT on 7B + 10k examples: order of magnitude cost comparison
You want to fine-tune a 7B Llama on ~10k high-quality SFT examples. Walk through the order of magnitude cost difference between (a) LoRA on a single H100 and (b) full fine-tuning. Mention the variants (QLoRA on a 24 GB consumer GPU) and how cost scales with model size.
LoRA on one H100 runs about 30-60 minutes for roughly $10-30. Full 7B fine-tuning needs 8 H100s and lands near $200-800, about 30 times more.
Picture a 7B model as a giant filing cabinet you want to retrain. Full fine-tuning rewrites every drawer, and to do that you must lay out a working copy of every file plus a second set of sticky-notes tracking how to nudge each one. That spread of paper needs eight desks, which here means eight GPUs, and most of the day. LoRA instead clips a tiny notepad onto the cabinet and only writes on the notepad, so the cabinet itself stays put. One desk holds the whole thing, and you finish before lunch. QLoRA shrinks the cabinet to a quarter its size first, so even a cheap home desk fits the job. Same training data either way, but the notepad approach costs a few dollars while rewriting every drawer costs hundreds.
Detailed answer & concept explanation~7 min readEverything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Everything important, quickly.
5 min: two cost levers, the memory math for full FT, why it forces 8 GPUs, LoRA single-H100 numbers, QLoRA on a 4090, and superlinear versus near-linear scaling.
| Aspect | LoRA (1× H100) | Full FT (8× H100) | QLoRA (1× 4090) |
|---|---|---|---|
| GPUs needed | 1 (80 GB) | 8 (sharded) | 1 (24 GB) |
| What is optimised | Tiny adapter | All 7B weights | Adapter on 4-bit base |
| Peak memory driver | Frozen base plus small adapter | Adam state on all params (~56 GB) | 4-bit base (~4 GB) plus adapter |
| Wall clock | 30-60 min | 6-12 hours | A few hours |
| All-in cost | $10-30 | $200-800 | $2-5 |
Real products, models, and research that use this idea.
- Unsloth publishes Colab notebooks fine-tuning Llama 3.1 8B with QLoRA on a free 16 GB T4, finishing a small SFT run for effectively zero cloud cost.
- Together.ai and Modal Labs both expose single-H100 LoRA jobs that complete 7B SFT runs in under an hour for single-digit dollar bills.
- The original QLoRA work demonstrated 4-bit fine-tuning of a 65B model on one 48 GB GPU, the result that made consumer-GPU full-quality adapters practical.
- OpenAI's fine-tuning API hides this entirely behind per-token pricing, billing roughly $25 per million training tokens for GPT-4o-mini-class adapters.
- Axolotl and LLaMA-Factory config templates default to 8×H100 FSDP for full 7B fine-tuning, matching the $200-800 per-run ballpark on rented clusters.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy does Adam optimiser state cost roughly four times the model size in memory?
QHow would you estimate wall-clock hours before launching the run?
Don't say thisRed flags and common mistakes that signal junior thinking. Click to expand.
Red flags and common mistakes that signal junior thinking. Click to expand.
Quoting one dollar figure without showing the memory math. The cost gap is downstream of optimiser state forcing full fine-tuning onto a multi-GPU cluster, which LoRA sidesteps entirely.
The night-before-the-interview bullets. Scan these on the way to the call.
Primary sources. Skim if you want the original framing.
Same topic, related formats. Practice these next.