**Vision-language models** are LLMs that can read images alongside text. Examples in 2026: GPT-5.5 and GPT-4o with image input, Claude Opus 4.7, Gemini 3.1 Pro. They look like chat models, but prompting them has three twists you do not see with text-only.
First, where you put the image relative to the text matters, and the right answer is not the same for every model. Some models do better if you describe the task first and then attach the image. Others handle image-first fine. You have to evaluate for your task.
Second, image resolution is not free. Each image is converted into a stack of tokens by tile-based encoding; a higher-resolution image becomes more tokens, costs more, and lets the model see more detail. A small thumbnail might cost a few hundred tokens; a large screenshot can cost a few thousand.
Third, if you pass more than one image, label them. 'Image 1 is the original, Image 2 is the edit' lets the model reference them unambiguously. Without labels, asking 'is the third one cropped correctly' is ambiguous.
**Three things change** when you move from text-only to vision-language prompting.
First, **image position** is a real prompt-engineering knob. Different providers and even different model versions handle text-first vs image-first ordering differently. Anthropic historically recommends putting images before the question that references them; OpenAI works fine either way for many tasks but exhibits per-task preferences. You evaluate the order on your own dataset rather than assume.
Second, **image resolution drives both cost and quality**. Providers tokenize images into a tile-based budget; a higher-resolution image becomes more image tokens, which raises cost and lets the model see fine detail. OpenAI exposes explicit 'low' vs 'high' detail levels with different token costs. Anthropic computes tokens from pixel-tile counts. The implication is that you should match resolution to the task: thumbnails for 'what is this' coarse classification, full resolution for fine-grained reading like dashboards or medical imagery.
Third, **multi-image prompts need explicit labeling**. Pass three images without labels and any later reference is ambiguous. Pass them as 'Image 1: original, Image 2: 3 days later, Image 3: 7 days later' and the model can answer questions about each one by reference.
Quick reference
TL;DR
Image position relative to text affects quality (model-dependent), image resolution drives both token cost and detail recovery, and multi-image prompts need explicit numbering to reference unambiguously.
Imagine giving someone a stack of photos and a question. If you hand them the photos first and then ask, they study the photos with the question in mind. If you ask first and then show photos, they look for the answer. Different listeners prefer different orders. Bigger, sharper photos take longer to study but you can see more detail. And if you show several photos, you have to say 'the one on the left' or 'photo two', or the listener will get confused about which one you mean. Vision-language models work the same way; the photos are images, the listener is the model, and the model has preferences about order, detail, and labeling.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Tasks where the image is purely decorative and the answer comes from text Sending the image at all wastes image tokens for no quality gain; strip it from the prompt or send it at the smallest available detail level.
Latency-critical paths under tight p99 budgets Image tokenization and projection add real per-call latency; a text-only fallback or a smaller model may meet SLA where the vision call cannot.
Tasks that need pixel-perfect output (object boundaries, exact coordinates) Vision-language models describe and reason about images; they are not segmentation or detection networks. Use SAM-style models for masks and CV pipelines for precise coordinates.
Quick scan · 2–4 min
Everything important, quickly.
4 min: how images become tokens + position vs attention + resolution vs cost + multi-image labeling + one per-task eval pattern.
Real-world examples
Real products, models, and research that use this idea.
1Anthropic's prompt-engineering docs recommend placing images before the question in Claude Opus 4.7 for tasks where the question references the image; matches their multi-modal prompt-engineering guidance.
2OpenAI exposes 'low' and 'high' detail levels for image input in GPT-5.5, with documented per-call token costs at each level.
3Medical imaging copilots (PathAI-style workflows, Aidoc) send the scan at full resolution because lesion detection depends on every pixel; thumbnail mode would cripple the diagnosis.
4Cursor and Copilot, when given UI screenshots, label them ('Screenshot 1: current state') so the model can reference each one in a multi-image debugging chat.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QIf your task is 'read this dashboard and summarize the three highest numbers,' which image resolution and position would you choose, and why?
Sign in or go Pro to see the approach for this follow-up.
QHow would you A/B test image position on your own data?
Sign in or go Pro to see the approach for this follow-up.
QWhat changes if you start streaming a video instead of single images?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Treating vision-language models like text models with images bolted on; image position, resolution, and labeling all change the answer in ways text-only prompts never see.