Once weights and decoding knobs are pinned, the prompt is the only inference-time input with an unbounded action space, so it carries almost all the controllable variance.
Imagine you bought a piano and it is already tuned. You cannot retune it mid-concert, and the volume pedal only goes between soft and loud. The one thing that still changes every performance is the sheet music you put in front of it. The piano is the model, the pedal is a decoding knob like temperature, and the sheet music is the prompt. The prompt can be any length, any structure, any examples, any rules. That is why people say prompt engineering is the main lever you have once the model is shipped, because the others are either locked or come with a tiny dial.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
There is a moment in most LLM projects where someone asks why prompt engineering gets so much attention when modern models seem to do well out of the box. The honest answer is structural, not aesthetic. Once a model is deployed, the function the model computes is fixed. Inference time is the regime where weights cannot move, and that fact alone reshapes what counts as a meaningful lever.
This question lives at the boundary between research framing and production framing. Researchers tend to think of inference as just sampling. Production teams think of inference as the only place where most quality decisions actually happen, because retraining is expensive and rare. The framing the prompt is the only lever is a compressed way of stating that asymmetry, but it deserves unpacking before you repeat it in an interview.
The deep dive walks through what is fixed, what is bounded, and what is unbounded. It then asks why the unboundedness matters in practice, and where the framing breaks down honestly.
What is fixed at inference time
When the request hits the model, the parameters θ are already baked. They came out of pretraining and any later fine-tuning rounds. From the runtime's point of view, the model is a deterministic function from input tokens to a next-token probability distribution. You cannot change that function without re-running training, which is by definition not an inference operation.
The practical consequence is that nothing the request itself does can move a single weight. You can swap to a different model, but that is choosing a different function, not adjusting the current one. This is why most production teams treat model selection as a deployment decision, not a request-time decision, even when they expose multiple models behind a router.
The takeaway from this section is simple. Weights are out of the conversation once the model is live. Whatever you change has to live somewhere else.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Anthropic Workbench exposes Claude Opus 4.7 with temperature and top-p sliders, but the central canvas is the prompt because that is where teams iterate.
- OpenAI Playground gives GPT-5.5 the same decoding sliders, but the documented optimization path for production traffic is prompt restructuring, not slider tuning.
What an interviewer would ask next. Try answering before peeking at the approach.
QIf decoding params are bounded, why do production teams still tune them?
Frame as calibration and cost: temperature for diversity tasks, top-p for tail control, max_tokens for latency and dollars per call.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Treating temperature and top-p as serious alternatives to prompt design, when they are scalar knobs with a tiny action space compared to an arbitrary-length input string.
60 second bullets to scan on the way to the call.
What is frozen at inference vs adjustable
Why decoding params have a small action space
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.