Skip fine-tuning when prompting or RAG already suffices, when knowledge changes often, when data is scarce, when you lack an eval harness, or when the base is already strong enough.
Fine-tuning is like sending an employee to an expensive multi-week training course. It permanently rewires how they work. You only do that when cheaper options have failed. If a sticky note on their desk fixes the problem, you write the note instead of booking the course. If the facts they need change every day, a note works but the course goes out of date instantly. If you have only three examples to train on, the course teaches the wrong habits. And if you cannot test whether the course actually helped, you have no way to know if you wasted the money. So before booking, you ask: would a note, a folder of references, or the employee's existing skills already do the job?
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Fine-tuning carries a gravitational pull in interviews and in real projects. It sounds like the serious, capital-F engineering answer, so candidates and teams reach for it as the default lever. The senior insight is the reverse: fine-tuning is the expensive last resort, and most of the time the correct move is to NOT do it.
This question is a decision filter. Six situations are listed, four are genuine reasons to avoid fine-tuning, and two are traps. The skill being tested is whether you can map each situation to the cheapest tool that actually solves it. The valid reasons fall out of a single mental model, an escalation ladder. You start with prompting, climb to retrieval, and only fine-tune when the rungs below have failed against a metric you trust.
The reasons to skip fine-tuning cluster into three buckets. The first is redundancy, where a prompt or RAG already does the job. The second is missing prerequisites, where you lack the data or the evaluation harness that make fine-tuning safe. The third is wrong-tool problems, where the need is formatting, tool access, or fresh knowledge that another mechanism owns. This deep dive walks each bucket, then dismantles the volume trap that catches junior candidates.
Fine-tuning is the top of an escalation ladder
The single most useful frame for this question is an ordered ladder of interventions. You climb only when the rung below has measurably failed.
The order is prompting, then retrieval, then fine-tuning. Prompting is free, instant to iterate, and reversible. You can rewrite a system prompt in seconds and ship it with no new infrastructure. Retrieval adds fresh and private knowledge without touching the model, and it stays current as your data changes. Fine-tuning sits at the top because it is slow, costly, and the hardest to undo. It permanently alters the weights and creates a new artifact you must version, evaluate, and maintain across model upgrades.
The cost asymmetry is the whole point. A prompt change costs minutes. A retrieval pipeline costs days of setup but pennies to update. A fine-tune costs data collection, a training run, an evaluation cycle, and ongoing maintenance every time the base model is deprecated. That maintenance tax is invisible on day one and brutal a year later, when the provider retires the base you tuned on and you must redo the whole run.
Every valid reason to NOT fine-tune is really a statement that a lower rung already solves the problem. That is why four of the six listed options are correct. Each describes a situation where prompting, retrieval, structured output, or tool-calling is the right tool, and reaching past it to fine-tuning only adds cost and risk. Internalise the ladder and the decision filter becomes almost mechanical.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
| Situation | Reach for instead | Why not fine-tune |
|---|---|---|
| Knowledge updates daily | RAG over a live index | Weights go stale the moment data changes |
| Fewer than ~500 clean examples | Few-shot prompting | Training overfits instead of generalising |
| Prompt already clears the golden bar | Keep the prompt | Marginal lift, real regression risk |
| Need strict JSON output | Structured-output or constrained decoding | Format is decoding, not a weight edit |
| Need database or tool access | Tool-calling and orchestration | Access is infrastructure, not weights |
| High request volume | Maybe distillation for cost | Volume is not a quality gap |
Real products, models, and research that use this idea.
- OpenAI's own fine-tuning guide tells teams to exhaust prompt engineering and retrieval before fine-tuning, treating it as a later-stage optimisation.
- Teams enforcing strict JSON on GPT-5.5 or Claude Opus 4.7 use native structured-output modes and libraries like Outlines rather than fine-tuning for schema adherence.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy does high request volume motivate distillation but not plain fine-tuning?
Separate quality from cost. Volume only raises per-call inference spend. Distillation trains a cheaper student to preserve quality at lower cost, which is an economics decision, not a capability gap fine-tuning would close.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Reaching for fine-tuning as the default lever. It is the expensive last resort. Prompting, RAG, and structured-output modes solve most problems first, and without an eval you cannot even tell if the fine-tune helped.
60 second bullets to scan on the way to the call.
Why fine-tuning is the last lever, not the first
When a prompt or RAG already suffices
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.