You're an ML engineer 6 months into a production LLM app. Prompt iteration has been productive but the last 5 iterations moved metrics under 2 points. Your team is considering fine-tuning. Walk through the decision: what signals confirm a real plateau (vs you've just run out of ideas), what gap shape analysis tells you whether fine-tuning or more prompt work is appropriate, and what concrete steps you'd take before committing to fine-tuning.
Confirm the plateau is structural, classify the remaining gap as behavior, knowledge, reasoning, or capability, then route each shape to its correct lever before committing to fine-tuning.
Imagine you have been tutoring a student for six months and their grades have stopped going up. Before deciding they need a totally different teacher, you ask two questions. First, have you actually tried every kind of practice problem, or just kept assigning the same kind? Second, what kind of mistakes are they still making? If they keep forgetting facts, hand them a textbook to look up answers (that is RAG). If they keep using the wrong tone in essays, give them more formal training (that is fine-tuning). If they cannot follow multi-step reasoning, teach them to slow down and check each step (that is self-consistency). And sometimes the honest answer is they are at the ceiling of what is possible right now, and you wait for a better tutor.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Plateau decisions are where most LLM teams either compound their leverage or burn months on the wrong project. The temptation to fine-tune is real because it feels like serious engineering work, and the deliverable is more visible than another round of prompt iteration. But the cost is large, the failure modes are subtle, and the wrong call is hard to walk back.
This is also where the seniority of the engineer shows. The junior move is to call any flat metric a plateau and start the fine-tune project. The senior move is to verify the plateau is real, diagnose the shape of the remaining gap, exhaust the cheaper levers, and only then commit to fine-tuning. Most teams who follow that discipline discover that the answer was never fine-tuning in the first place.
The deep dive walks through the three-stage decision: confirming the plateau, classifying the gap, and exhausting the cheaper levers. It also covers the production reality that fine-tuning is the right answer roughly 20-30% of the time when teams think it is.
Confirming the plateau is real
A real plateau has three signatures that distinguish it from stagnation. The first is engineering redundancy. If only one person has iterated, the plateau is probably the engineer's blind spot, not the model's ceiling. Bring in a second engineer for a sprint and see whether they hit the same ceiling from a different angle.
The second is model redundancy. Run the current best prompt against two or three frontier models in the same tier: Claude Sonnet 4.x, GPT-5.5, Gemini 3.1 Flash. If they all converge to roughly the same metric, the ceiling is structural to the prompt strategy. If one model significantly outperforms the others, you are looking at a model-specific limitation, not a prompt-design plateau.
The third is failure-mode clustering. Pull 50 recent failures and tag them. If they cluster on the same one or two failure modes across iterations, the plateau is real and you have a clear target for the next lever. If failures scatter across modes, the prompt is leaking on multiple dimensions and you have not yet maximized any single one. Multiple-dimension leaks are a sign of stagnation, not a true plateau.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Anthropic's prompt-engineering documentation explicitly recommends exhausting structural variants like prompt chaining and tool use before fine-tuning Claude Opus 4.7.
- Cursor's production stack ships hand-tuned system prompts on Claude Opus 4.7 and GPT-5.5 with no per-model fine-tune, validating that prompt iteration alone covers most code edit failure modes.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you actually classify a remaining gap into behavior vs knowledge?
Show me the failure: if a knowledgeable human would say I needed to look that up, it is knowledge; if they would say the model knew but said it wrong, it is behavior.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Calling a plateau real after one engineer iterated for a sprint, then committing to a fine-tune that solves the wrong problem because the gap was actually knowledge.
60 second bullets to scan on the way to the call.
Three signatures of a real plateau
Behavior vs knowledge vs reasoning vs capability gap classification
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.