Defend the claim that full fine-tuning still beats LoRA for some 2026 production workloads

Short answer·Medium·4.0 · 0·~3 min·Relevant atDatabricks Meta OpenAI

Attempt it

LoRA and QLoRA dominate the headlines in 2026, yet a sober production team would still pick full fine-tuning in specific situations. Make the case: when does full FT genuinely beat LoRA, and what is the actual quality gap?

Free · 2 AI evals / day

**Full fine-tuning** updates every weight in the model. **LoRA** trains a small low-rank correction and leaves the base weights frozen. LoRA is cheaper in nearly every dimension: less GPU memory, smaller checkpoints, fast to swap. So why would anyone still pick the expensive option? Three conditions push the decision toward full fine-tuning. First, lots of data. If you have 10,000 high-quality examples or more, the extra capacity of full fine-tuning is actually used. With small datasets the extra capacity becomes an overfitting vector. Second, big distribution shift. If your domain is far from the base model's pretraining mix, like legal, biomedical, or a non-English script, every weight needs to move a little. A low-rank correction only changes the model along a few directions, which is fine for style nudges but cramped for a genuine domain shift. Third, simple serving. LoRA's selling point at inference is adapter portability, the ability to swap several adapters against one frozen base. If you serve a single product or a single tenant, you never use that feature, so full fine-tuning's lack of it costs you nothing. The quality gap on hard benchmarks is usually small, a few points, and DoRA narrows it further. But on the right workload, that small gap is worth the extra cost.

**Full fine-tuning still wins** in a narrow but real slice of production. Three conditions, each independent, push the decision in that direction. The first is dataset size. With ten thousand or more high-quality examples, the extra capacity of full fine-tuning is actually used. Below that, the extra weights become an overfit vector and LoRA's regularising low-rank constraint is a feature, not a limit. The second is distribution shift. Vanilla LoRA expresses the update as a sum of low-rank corrections, which is enough for instruction-style polish on a base model that already knows your domain. When the target distribution sits far from the base, like Japanese legal contracts, biomedical literature, or a low-resource language, the update needs to touch every weight and the low-rank constraint becomes a ceiling. Full fine-tuning takes that ceiling off. The third is serving topology. LoRA's headline serving win is adapter portability. One frozen base, dozens of adapters swapped per request, kilobytes per skill. If you run a single tenant or a single product, you never use that feature, and the LoRA-versus-full trade collapses to pure quality. The measured quality gap is usually small, a couple of points on MMLU-medical, LegalBench, or domain-specific reasoning tasks. DoRA narrows it further by giving LoRA a separate magnitude knob, and the strongest 2026 variants land within a point of full fine-tuning. But on a large, far-from-base, single-tenant workload, that small gap is worth the cost.

Quick reference

TL;DR

Full fine-tuning still wins when the dataset is large, the distribution shift from base is broad, and the serving stack is single-tenant. The gap on hard benchmarks is small but real.

Memory aid

Easy to grasp

Imagine a master chef trained on global cuisine, and you want them to cook only Japanese kaiseki. If you have just a few recipes to teach them, sticking a small notebook of tips in their pocket works fine: that is the popular cheap method. But if you have thousands of recipes and the cuisine is genuinely different from anything they trained on, you actually want them to relearn their habits across every dish, not just glance at a notebook. The full retraining is slower and more expensive, and it only pays off when you have lots of teaching material, the new style is genuinely far from the old, and you serve only this one restaurant. In every other case the notebook is a better deal.

Key concepts

Full fine-tuning LoRA adapters DoRA Method selection

Detailed answer & concept explanation

~7 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Quick scan · 2–4 min

Everything important, quickly.

5 min: the three conditions for full FT, the capacity argument behind low-rank ceilings, the adapter-portability trade, where the measured gap appears, how DoRA narrows it, and the production decision framework.

Side-by-side

Axis	Full fine-tuning	LoRA / DoRA
Training memory	16+ bytes per param with Adam	Low single-digit GB for adapters
Quality ceiling	Highest on far-from-base domains	Within ~1 to 4 pts on most benchmarks
Checkpoint size	Full base weights per fine-tune	Megabytes per adapter
Adapter portability	None, one model per task	Many adapters per frozen base
Best regime	Large data, big shift, single tenant	Small data, near-base shift, multi-tenant serving
Overfit risk on small data	High, extra capacity hurts	Low, low-rank constraint regularises

Real-world examples

Real products, models, and research that use this idea.

BloombergGPT trained a 50B model fully on financial corpora because the distribution shift from general web text was large enough to justify the cost.
Med-PaLM 2 used full fine-tuning over medical literature where domain accuracy was the product, not a polish step.
Mistral Large 3 customer deployments at law firms tend to choose full fine-tuning on tens of thousands of contract examples for single-tenant legal assistants.
Llama 4 Maverick instruction-tuned releases now ship LoRA-tuned variants as the open community default, while frontier closed-tenancy deployments still run full fine-tunes when data warrants.
DeepSeek V4 community fine-tunes use DoRA for low-resource languages where vanilla LoRA closes most but not all of the gap to full fine-tuning.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QOn a 10k-example legal contract dataset, how would you decide whether LoRA r=64 plus DoRA is enough or you need full fine-tuning?

QWhy does the quality gap show up more on reasoning and dense factual recall than on instruction following?

QIf your serving stack hosts twenty per-customer skills today and you fine-tune fully, what changes in your inference architecture?

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Declaring full fine-tuning obsolete in 2026 because LoRA is cheaper. It is cheaper everywhere, but not always close enough in quality on large datasets that sit far from the base model to ignore the gap.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

References

Primary sources. Skim if you want the original framing.