At inference time, once the model and decoding parameters are pinned, why is the prompt described as the ONLY meaningful lever for changing LLM output?

At inference time, three things shape the output: the model weights, the decoding parameters, and the prompt. The weights are baked in. You cannot change them without fine-tuning, which is a separate training step, not a runtime knob. Decoding parameters such as temperature, top-p, and max tokens are tunable at request time, but each one is a single small number. Temperature lives in roughly the [0, 2] range. Top-p lives in (0, 1]. They control how the model samples from its next-token distribution, but they cannot change what the model actually thinks about. The prompt is different. It is a string of any length, with any content, in any structure. You can add examples, rules, formatting instructions, retrieved documents, and explicit reasoning scaffolds. That makes its action space effectively unbounded, which is why it carries almost all the meaningful variation in output once the model is fixed.

Three inputs feed the model at inference: weights, decoding params, and the prompt. Weights are fixed once the model ships, unless you fine-tune, and that is not an inference operation. Decoding params are real knobs but each is a scalar with a tiny action space. Temperature is one number in roughly [0, 2]. Top-p is one number in (0, 1]. They reshape sampling but cannot change what the model reads. The prompt is structurally different. It is an arbitrary string, so its space of possible values is effectively infinite. You can change role assignment, examples, reasoning scaffolds, format constraints, and retrieved context. Each of those rewrites WHAT the model attends to, not just how it samples. That is why a senior engineer treats prompt engineering as the highest-leverage inference-time surface. Decoding params are calibration, the prompt is content. The honest framing is that the prompt is not the only lever, it is the only lever with unbounded reach, which is why almost all controllable output variance lives there.