When should you fine-tune the model instead of iterating on the prompt?

MCQ·Medium·4.0 · 0·~1 min·Asked atDify Intuit Palantir·Relevant atAnthropic Databricks OpenAI

Attempt it

Prompt engineering and fine-tuning both change how an LLM (large language model) behaves, but they work at different layers. A prompt is the text you send at request time. Fine-tuning is a training step that updates the model weights so the new behavior is baked in. Prompts are the right starting point. They are fast to iterate, free of training cost, and easy to change. Many production LLM apps ship with prompt engineering only, because modern models are capable enough that a good prompt covers most of the gap. Fine-tuning becomes the right tool when prompt iteration has stopped moving the metric. It also fits when you need the same tone or output format every time. A third case is a prompt so long that token cost dominates. Do not reach for fine-tuning when the real gap is missing knowledge. Fine-tuning does not reliably teach facts. For fresh, private, or frequently changing knowledge, RAG (retrieving documents at request time) is the correct lever. The answer is B. A is wrong: prompts run in production every day. C is wrong: no token count decides it, and prompt caching cuts long-prompt cost. D is wrong for the knowledge reason above.

Prompt for behavior changes you can describe in a handful of examples. Fine-tune for behavior changes you cannot reliably enforce in-context. Use RAG for knowledge. The decision rubric has three triggers for fine-tuning. First, prompt iteration has plateaued: you have tried structural variants (decomposition, self-consistency, few-shot ordering, schema directives) and the metric will not move further. Second, you need behavioral consistency across many contexts: tone, format, refusal patterns, or strict schema adherence that the model occasionally violates under prompt-only instruction. Third, the prompt has become so long that per call cost dominates, and amortizing the instructions into weights is cheaper than caching. Stay on prompts when tasks rotate often, when iteration speed is the bottleneck, or when you lack 100-1000 high-quality input-output pairs to train on. LoRA and QLoRA have made fine-tuning much cheaper than it was a few years ago. But the data and eval discipline are still the gating constraint. Prompting and fine-tuning are not opposites. A common production pattern is to fine-tune the base behavior, then prompt-engineer the task layer on top. That makes B the correct option. A is wrong because prompts run in production every day. C is wrong because no fixed token count is the trigger, and prompt caching cuts long-prompt cost. D is wrong because fine-tuning suits behavior, while new knowledge goes to RAG.

TL;DR

Fine-tune for stable behavior at scale; prompt for fast iteration and changing tasks; use RAG for fresh or private knowledge.

Memory aid

Easy to grasp

Imagine you are training a new hire. You can write them a detailed instruction sheet (a prompt) that they read fresh every morning, or you can send them to a two-week bootcamp (fine-tuning) so the instructions become muscle memory. The sheet is flexible because you can rewrite it tonight. But the new hire has to re-read it every single day, which is slow at scale. The bootcamp is fast at runtime because they just know what to do, but updating their training means another bootcamp. And if the question is about facts they have not learned, neither helps. You hand them a reference book to look things up, which is the RAG pattern.

Key concepts

Prompt Engineering Fine-tuning Retrieval-Augmented Generation

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Prompt engineering and fine-tuning both adapt a base model toward a target behavior. But they sit at different layers in the stack and have different cost curves. Keep both in mind and reach for whichever lever matches the actual gap. Reaching for the wrong one wastes weeks. The usual wrong choice is fine-tuning when the real problem was missing facts or insufficient prompt iteration.

The lever choice shapes the whole engineering plan. Fine-tuning means a data curation team, a training pipeline, an eval harness for pre/post comparison, and a re-tune cadence. Prompt engineering means a versioned prompt library, a prompt eval CI, and a faster but more frequent iteration loop. The teams look different, the budgets look different, and the failure modes look different.

Cheaper adapters (LoRA, QLoRA) and prompt caching have shifted the cost picture, but the underlying decision rubric is mostly the same.

What each lever changes

Prompt engineering changes the input to a fixed function. The weights stay frozen. You are providing in-context instructions, examples, and constraints that condition the model's existing capabilities toward a specific behavior. Nothing about the model itself moves.

Fine-tuning changes the function. A training pass on a curated dataset updates the weights (full fine-tune) or a small set of low-rank adapters (LoRA, QLoRA). After the run, the model behaves differently even with no instructions, because the new behavior is now compiled into the parameters.

The practical consequence is that prompts are reversible in a minute and fine-tunes are not. A bad prompt change is a git revert. A bad fine-tune means retraining or rolling back to a previous adapter, which is hours of work and a new eval pass. This asymmetry is why production teams prompt first and fine-tune only when prompts have provably stopped moving the metric.

When prompt engineering is the right call

When fine-tuning is the right call

The knowledge trap

The cost picture and how to combine the levers

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Concern	Prompt engineering	Fine-tuning
Iteration speed	Minutes; live edits and reverts	Hours to days; a training run per change
Cost per change	Near zero	Tens to hundreds of USD with LoRA; more for full fine-tune
Best for	Task-level behavior, format guidance, reasoning scaffolds	Stable tone, strict schema, long stable instructions
Knowledge updates	RAG via retrieved context	Poor fit; weights do not reliably install facts
Production discipline	Versioned prompts + eval CI	Versioned adapters + golden-set evals pre/post

Real products, models, and research that use this idea.

Many chat and coding products call a hosted general model with a different system prompt per feature, so a behavior change ships in minutes with no training run.
Teams that need a strict output schema or a fixed brand tone often train a LoRA adapter on an open-weight model. LoRA trains small low-rank matrices, so the adapter file stays small and cheap to swap.

What an interviewer would ask next. Try answering before peeking at the approach.

QHow does LoRA change the prompt vs fine-tune decision?

Frame as a cost shift, not a behavior shift: training got cheaper, but data quality and eval discipline are still the gating constraints.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Reaching for fine-tuning when the gap is actually missing knowledge; that is a RAG problem, and weight updates will not reliably add facts.

60 second bullets to scan on the way to the call.

Three triggers that justify fine-tuning
Why knowledge gaps route to RAG

Primary sources. Browse if you want the original framing.