When should you fine-tune the model instead of iterating on the prompt?

Prompt engineering and fine-tuning both change how an LLM behaves, but they work at different layers. A prompt is the text you send at request time. Fine-tuning is a training step that updates the model weights so the new behavior is baked in. Prompts are the right starting point. They are fast to iterate, free of training cost, and easy to change when the task changes. Most production LLM apps in 2026 ship with prompt engineering only, because models like Claude Opus 4.7 and GPT-5.5 are capable enough that a well-written prompt covers most of the gap. Fine-tuning becomes the right tool when prompt iteration has stopped moving the metric, when you need the same tone or output format every single time across thousands of contexts, or when the prompt has grown so long that token cost is the dominant problem. The trap is reaching for fine-tuning when the real gap is missing knowledge. Fine-tuning does not reliably teach facts. For fresh, private, or frequently changing knowledge, RAG is the correct lever.

Prompt for behavior changes you can describe in a handful of examples. Fine-tune for behavior changes you cannot reliably enforce in-context. Use RAG for knowledge. The decision rubric has three triggers for fine-tuning. First, prompt iteration has plateaued: you have tried structural variants (decomposition, self-consistency, few-shot ordering, schema directives) and the metric will not move further. Second, you need behavioral consistency across many contexts: tone, format, refusal patterns, or strict schema adherence that the model occasionally violates under prompt-only instruction. Third, the prompt has become so long that per-call cost dominates, and amortizing the instructions into weights is cheaper than caching. Stay on prompts when tasks rotate often, when iteration speed is the bottleneck, or when you lack 100-1000 high-quality input-output pairs to train on. LoRA and QLoRA make fine-tuning cheaper in 2026 than it was three years ago, but the data and eval discipline are still the gating constraint. The honest framing is that prompting and fine-tuning are not opposites. Most production stacks fine-tune the base behavior, then prompt-engineer the task layer on top.