LoRA vs full FT on 7B + 10k examples: order of magnitude cost comparison
You want to fine-tune a 7B Llama on ~10k high quality SFT examples. Walk through the order of magnitude cost difference between (a) LoRA on a single H100 and (b) full fine-tuning. Mention the variants (QLoRA on a 24 GB consumer GPU) and how cost scales with model size.
LoRA on one H100 runs about 30-60 minutes for roughly $10-30. Full 7B fine-tuning needs 8 H100s and lands near $200-800, about 30 times more.
Picture a 7B model as a giant filing cabinet you want to retrain. Full fine-tuning rewrites every drawer, and to do that you must lay out a working copy of every file plus a second set of sticky-notes tracking how to nudge each one. That spread of paper needs eight desks, which here means eight GPUs, and most of the day. LoRA instead clips a tiny notepad onto the cabinet and only writes on the notepad, so the cabinet itself stays put. One desk holds the whole thing, and you finish before lunch. QLoRA shrinks the cabinet to a quarter its size first, so even a cheap home desk fits the job. Same training data either way, but the notepad approach costs a few dollars while rewriting every drawer costs hundreds.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Estimating fine-tuning cost on H100s is a memory problem wearing a dollar-sign costume. The number of GPUs you need is decided almost entirely by peak memory, and the number of GPUs times the hours times the hourly rate is the bill. So the moment you can do the memory arithmetic in your head, you can quote a credible cost band without ever launching a job.
The scenario here is concrete: a 7B Llama, 10k high-quality SFT examples, three epochs. That is the canonical interview setup because it sits right on the boundary where the choice of method changes the answer by more than an order of magnitude. The same data, the same model, the same hardware family, and yet LoRA costs you a coffee while full fine-tuning costs you a weekend cluster.
The reason is not the forward and backward passes; both methods push similar token counts through similar matmuls. The reason is what each method has to keep resident in GPU memory. Full fine-tuning has to hold an optimiser state for every one of the 7 billion parameters. LoRA freezes the base and optimises a few hundred MB of adapter. That single difference cascades into GPU count, wall clock, and ultimately the price.
This deep dive does the arithmetic end to end. It walks the two cost levers, the memory breakdown that decides how many GPUs you need, the throughput to hours estimate, plausible 2026 hourly rates for both H100 and consumer cards, and how the whole picture scales when the model grows from 7B to 70B. By the end you should be able to defend a cost band on a whiteboard from first principles, which is exactly what the interviewer is testing.
The two cost levers: GPU-hours and GPU-count
Every fine-tuning bill reduces to one expression. You rent some number of GPUs, for some number of hours, at some hourly rate. The order of magnitude cost formula is just this:
The wall-clock hours come from total training tokens divided by throughput. For this scenario, 10k examples at roughly 1k tokens each over 3 epochs is about 30M training tokens. A 7B model with packing and FlashAttention runs at perhaps 50-80k tokens per second on an H100, which is 6-10 minutes per epoch.
The number of GPUs, though, is not a throughput choice. It is a memory choice. If the job fits one card, you rent one card. If peak memory exceeds the card, you are forced to shard across several, and now you pay for all of them plus a communication tax that lowers per-GPU throughput. The entire LoRA-versus-full-FT cost gap lives in this second lever.
This is why the rate alone tells you almost nothing. A single H100 at $3 per hour and an 8-GPU cluster at $24 per hour are the same per-card rate, yet the cluster run costs roughly an order of magnitude more, because it burns eight cards for longer. The hourly number is fixed by the market; the GPU count and the hours are what your method actually controls.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
| Aspect | LoRA (1× H100) | Full FT (8× H100) | QLoRA (1× 4090) |
|---|---|---|---|
| GPUs needed | 1 (80 GB) | 8 (sharded) | 1 (24 GB) |
| What is optimised | Tiny adapter | All 7B weights | Adapter on 4-bit base |
| Peak memory driver | Frozen base plus small adapter | Adam state on all params (~56 GB) | 4-bit base (~4 GB) plus adapter |
| Wall clock | 30-60 min | 6-12 hours | A few hours |
| All-in cost | $10-30 | $200-800 | $2-5 |
Real products, models, and research that use this idea.
- Unsloth publishes Colab notebooks fine-tuning Llama 3.1 8B with QLoRA on a free 16 GB T4, finishing a small SFT run for effectively zero cloud cost.
- Together.ai and Modal Labs both expose single-H100 LoRA jobs that complete 7B SFT runs in under an hour for single-digit dollar bills.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy does Adam optimiser state cost roughly four times the model size in memory?
Walk through bf16 weights plus bf16 gradients plus the two fp32 Adam moments per parameter. The moments dominate, and they are exactly the state LoRA never has to allocate on the frozen base.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Quoting one dollar figure without showing the memory math. The cost gap is downstream of optimiser state forcing full fine-tuning onto a multi-GPU cluster, which LoRA sidesteps entirely.
60 second bullets to scan on the way to the call.
GPU-hours and GPU-count as the two cost levers
Memory breakdown of weights, gradients, and Adam state
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.