Image position relative to text affects quality (model-dependent), image resolution drives both token cost and detail recovery, and multi-image prompts need explicit numbering to reference unambiguously.
Imagine giving someone a stack of photos and a question. If you hand them the photos first and then ask, they study the photos with the question in mind. If you ask first and then show photos, they look for the answer. Different listeners prefer different orders. Bigger, sharper photos take longer to study but you can see more detail. And if you show several photos, you have to say 'the one on the left' or 'photo two', or the listener will get confused about which one you mean. Vision-language models work the same way; the photos are images, the listener is the model, and the model has preferences about order, detail, and labeling.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Vision-language models look like chat models with a new input type, but prompting them well is a real shift from text-only prompt engineering. The same three properties that make text prompts work, position, length, structure: all reappear in image-modified forms with their own gotchas.
The deep dive below works through how images actually become tokens inside a vision-language model, why position matters and why it varies by model, how image resolution trades against cost and quality, and why multi-image prompts fall apart without explicit labels. The closing section names the per-task eval pattern that separates serious multi-modal teams from teams who just enabled the image input checkbox.
How images become tokens inside the model
A vision-language model is, under the hood, an LLM with a vision encoder bolted onto the front. The encoder (typically a ViT-style patch transformer) chops the image into a grid of patches, embeds each patch into the same dimensional space as text tokens, and emits a sequence of image tokens. Those image tokens get spliced into the LLM's token stream wherever you put the image in the message. From the LLM's perspective, an image is just a contiguous block of tokens it attends over, exactly like a paragraph of text.
This is why classical text prompt engineering rules carry over. Lost-in-the-middle attention still applies to image tokens. Recency bias still applies. Cache-prefix logic still applies. The image is not a special object; it is a long span of tokens that happens to encode pixels rather than language. Most multi-modal prompt design follows from that fact.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Anthropic's prompt-engineering docs recommend placing images before the question in Claude Opus 4.7 for tasks where the question references the image; matches their multi-modal prompt-engineering guidance.
- OpenAI exposes 'low' and 'high' detail levels for image input in GPT-5.5, with documented per-call token costs at each level.
What an interviewer would ask next. Try answering before peeking at the approach.
QIf your task is 'read this dashboard and summarize the three highest numbers,' which image resolution and position would you choose, and why?
High detail (numbers are fine print; pixels matter). Image-first so the model can scan before reading the instruction. Justify each choice in terms of attention and token cost.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Treating vision-language models like text models with images bolted on; image position, resolution, and labeling all change the answer in ways text-only prompts never see.
60 second bullets to scan on the way to the call.
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.