You're an ML engineer 6 months into a production LLM app. Prompt iteration has been productive but the last 5 iterations moved metrics under 2 points. Your team is considering fine-tuning. Trace the decision: what signals confirm a real plateau (vs you've just run out of ideas), what gap shape analysis tells you whether fine-tuning or more prompt work is appropriate, and what concrete steps you'd take before committing to fine-tuning.
Confirm the plateau is structural, classify the remaining gap as behavior, knowledge, reasoning, or capability, then route each shape to its correct lever before committing to fine-tuning.
Imagine you have been tutoring a student for six months and their grades have stopped going up. Before deciding they need a totally different teacher, you ask two questions. First, have you actually tried every kind of practice problem, or just kept assigning the same kind? Second, what kind of mistakes are they still making? If they keep forgetting facts, hand them a textbook to look up answers (that is RAG). If they keep using the wrong tone in essays, give them more formal training (that is fine-tuning). If they slip on multi-step problems, have them solve each problem several times and keep the answer that comes up most (that is self-consistency). And sometimes the honest answer is they are at the ceiling of what is possible right now, and you wait for a better tutor.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
At a plateau, a team either picks the right next lever or burns months on the wrong project. The temptation to fine-tune is real because it feels like serious engineering work, and the deliverable is more visible than another round of prompt iteration. But the cost is large, the failure modes are subtle, and the wrong call is hard to walk back.
A common mistake is to call any flat metric a plateau and start the fine-tune project. The better path is to verify the plateau is real, diagnose the shape of the remaining gap, exhaust the cheaper levers, and only then commit to fine-tuning. Many teams who follow that discipline discover that the answer was never fine-tuning in the first place.
The decision has three stages: confirm the plateau, classify the gap, and exhaust the cheaper levers.
Confirming the plateau is real
A real plateau has three signatures that distinguish it from stagnation. The first is engineering redundancy. If only one person has iterated, the plateau is probably the engineer's blind spot, not the model's ceiling. Bring in a second engineer for a sprint and see whether they hit the same ceiling from a different angle.
The second is model redundancy. Run the current best prompt against two or three frontier models in the same tier, ideally from different providers. If they all converge to roughly the same metric, the ceiling is structural to the prompt strategy. If one model significantly outperforms the others, you are looking at a model-specific limitation, not a prompt design plateau.
The third is failure-mode clustering. Pull 50 recent failures and tag them. If they cluster on the same one or two failure modes across iterations, the plateau is real and you have a clear target for the next lever. If failures scatter across modes, the prompt is falling short on multiple dimensions and you have not yet maximized any single one. Failures spread across many dimensions are a sign of stagnation, not a true plateau.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Anthropic recommends clear success criteria and an empirical test before prompt work. It notes that some failing evals are solved more easily another way, such as by choosing a different model.
- OpenAI recommends setting up evals before investing in fine-tuning. It also recommends rethinking the task or prompt if a first run on about 50 training examples shows no improvement.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you actually classify a remaining gap into behavior vs knowledge?
Read the failures: if a knowledgeable human would say "I needed to look that up," it is knowledge; if they would say the model knew but said it wrong, it is behavior.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Calling a plateau real after one engineer iterated for a sprint, then committing to a fine-tune that solves the wrong problem because the gap was actually knowledge.
60 second bullets to scan on the way to the call.
Three signatures of a real plateau
Behavior vs knowledge vs reasoning vs capability gap classification
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.