Skip fine-tuning when prompting or RAG already suffices, when knowledge changes often, when data is scarce, when you lack an eval harness, or when the base is already strong enough.
Fine-tuning is like sending an employee to an expensive multi-week training course. It permanently rewires how they work. You only do that when cheaper options have failed. If a sticky note on their desk fixes the problem, you write the note instead of booking the course. If the facts they need change every day, a note works but the course goes out of date instantly. If you have only three examples to train on, the course teaches the wrong habits. And if you cannot test whether the course helped, you have no way to know if you wasted the money. So before booking, you ask: would a note, a folder of references, or the employee's existing skills already do the job?
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Fine-tuning carries a gravitational pull in real projects. It sounds like the serious engineering answer, so teams reach for it as the default lever. The reality is the reverse: fine-tuning is the expensive last resort, and most of the time the correct move is to NOT do it.
Six situations are listed: five are genuine reasons to avoid fine-tuning, and one is a common misconception. Map each situation to the cheapest tool that solves it. The valid reasons come from a single mental model, an escalation ladder. You start with prompting, climb to retrieval, and only fine-tune when the rungs below have failed against a metric you trust.
The reasons to skip fine-tuning cluster into three buckets. The first is redundancy, where a prompt or RAG already does the job. The second is missing prerequisites, where you lack the data or the evaluation harness that make fine-tuning safe. The third is wrong tool problems, where the need is formatting or tool access that another mechanism owns.
Fine-tuning is the top of an escalation ladder
The most useful frame here is an ordered ladder of interventions. You climb only when the rung below has measurably failed.
The order is prompting, then retrieval, then fine-tuning. Prompting is free, instant to iterate, and reversible. You can rewrite a system prompt in seconds and ship it with no new infrastructure. Retrieval adds fresh and private knowledge without touching the model, and it stays current as your data changes. Fine-tuning sits at the top because it is slow, costly, and the hardest to undo. It permanently alters the weights and creates a new artifact you must version, evaluate, and maintain across model upgrades.
The cost asymmetry is the whole point. A prompt change costs minutes. A retrieval pipeline costs days of setup but pennies to update. A fine-tune costs data collection, a training run, an evaluation cycle, and ongoing maintenance every time the base model is deprecated. That maintenance cost is invisible on day one and brutal a year later. It hits when the provider retires the base you tuned on and you must redo the whole run.
Every valid reason to NOT fine-tune is a statement that a lower rung already solves the problem. That is why five of the six listed options are correct. Each describes a situation where prompting, retrieval, structured output, or tool calling is the right tool. Reaching past it to fine-tuning only adds cost and risk. With the ladder, the decision filter becomes almost mechanical.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
| Situation | Reach for instead | Why not fine-tune |
|---|---|---|
| Knowledge updates daily | RAG over a live index | Weights go stale the moment data changes |
| Fewer than ~50 clean examples | Few-shot prompting | Training overfits instead of generalising |
| Prompt already clears the golden bar | Keep the prompt | Marginal lift, real regression risk |
| Need strict JSON output | Structured-output or constrained decoding | Format is decoding, not a weight edit |
| Need database or tool access | Tool-calling and orchestration | Access is infrastructure, not weights |
| High request volume | Maybe distillation for cost | Volume is not a quality gap |
Real products, models, and research that use this idea.
- OpenAI says prompt engineering may be all you need, calls fine-tuning time-consuming, and advises setting up evals before investing in it. OpenAI is also winding down its fine-tuning platform.
- Teams that need strict JSON from hosted APIs use the provider's structured-output mode where it exists, and constrained decoding libraries like Outlines for open models, rather than fine-tuning for schema adherence.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy does high request volume motivate distillation but not plain fine-tuning?
Separate quality from cost. Volume only raises per-call inference spend. Distillation trains a cheaper student to preserve quality at lower cost, which is an economics decision, not a capability gap fine-tuning would close.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Reaching for fine-tuning as the default lever. It is the expensive last resort. Prompting, RAG, and structured-output modes solve most problems first, and without an eval you cannot even tell if the fine-tune helped.
60 second bullets to scan on the way to the call.
Why fine-tuning is the last lever, not the first
When a prompt or RAG already suffices
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.