How does a text prompt actually steer a text to image diffusion model?

Generating an image with diffusion starts from pure noise and removes a bit of it over many steps until a picture appears. The prompt has to steer that process the whole way. First the words go through a **text encoder** that turns them into a sequence of embeddings. Then, at each denoising step, the network that predicts the noise looks at those embeddings through cross-attention. So the prompt is read again and again, shaping every step toward matching the description rather than just nudging the start. The strength of that obedience is set by **classifier-free guidance**. The model makes two predictions, one with the prompt and one without, then pushes the result in the direction the prompt adds. A scalar called the guidance scale controls how far it pushes. A high guidance scale makes the image follow the prompt literally but can look forced and less varied. A low scale stays natural and diverse but drifts from the exact words. That scale is the knob you tune.

A text to image diffusion model generates by starting from noise and iteratively denoising a latent over many steps. The prompt steers this through two distinct mechanisms. First, conditioning. The prompt is run through a **text encoder** into a sequence of embeddings. At each denoising step, the noise-prediction network attends to those embeddings via cross-attention — the image latent supplies the queries, the text embeddings supply the keys and values. Because this happens at every step, the prompt continuously shapes how noise is removed, pulling the latent toward a configuration that matches the description rather than just seeding it once. Second, strength. **Classifier-free guidance** controls how hard the image obeys the text. At each step the model runs the network twice: once with the prompt and once with an empty prompt. It then extrapolates from the unconditioned prediction in the direction the prompt adds. The guidance scale sets how far. $$\hat{\epsilon} = \epsilon_\varnothing + s \cdot (\epsilon_{\text{cond}} - \epsilon_\varnothing)$$ The guidance scale s is the adherence knob. Push it high and the image follows the prompt literally but loses diversity and can look oversaturated. Keep it low and the image stays natural but drifts from the words. Typical values land around 5 to 9.