Where should an image be placed relative to the text that asks about it, and why does this matter for multimodal models?

Flashcard·Medium·4.0 · 0·~30s·Asked atFlipkart Turing Whylabs

Attempt it

Place the image immediately adjacent to the text that refers to it, with the image first and the question right after. Two reasons. First, the question reads as a continuation of what the model just saw, recency works for vision tokens the same way it works for text. Second, when there are multiple images the model can be confused about which one a later sentence refers to; co-locating each image with its question removes the ambiguity. Across providers the exact tokenization of images differs, but the locality principle is the same: keep the question and its image together, not separated by a wall of other content.

Your answer

The short answer: **place the image immediately before the text that asks about it**, and keep them adjacent. There are two reasons this matters. **Reason 1: recency.** Modern multimodal models (Claude Opus 4.7, GPT-5.5, Gemini 3.1 Pro) attend to nearby content more reliably than to distant content, the same lost-in-the-middle effect that applies to text applies to vision tokens. Putting the image right before the question makes the question read as a natural continuation of what the model just saw. **Reason 2: reference ambiguity.** When you send a single image and a single question, the model can figure out the binding even if the image is at the top and the question is at the bottom. When you send three images and three questions, the binding becomes ambiguous unless each image is co-located with its question. The model might attend to the wrong image when answering, especially if the images look similar. The pattern that works: ``` [image_1] Question about image 1. [image_2] Question about image 2. [image_3] Question about image 3. ``` The pattern that fails: ``` [image_1] [image_2] [image_3] Question about the first image. Question about the second image. Question about the third image. ``` In the second pattern the model has to do extra work to bind 'first' to `image_1`, and it gets it wrong more often than the co-located version. The exact image tokenization differs across providers, but the locality principle is universal. Keep each image next to the text that refers to it.

**The image goes immediately before the text that asks about it, and the two stay adjacent in the prompt.** This is the universal recommendation across 2026 multimodal providers (Anthropic, OpenAI, Google) for a reason that has both a positional component and a reference-resolution component. **The positional component is recency.** Multimodal models are subject to the same kind of positional attention bias that applies to text-only models. Content near the end of the input is attended to more reliably than content in the middle. When the image is placed right before the question, the question reads as a continuation of what the model just saw, the vision tokens are still recent in the attention pattern when the question begins. When the image is placed at the start of the prompt and the question is far below, the model has to look back across a wall of other content to ground its answer, and quality degrades. **The reference-resolution component matters most with multiple images.** When a prompt contains one image and one question, the binding is unambiguous: there is only one image to refer to. When a prompt contains three images and three questions, the binding has to be derived from the text. 'The first image' and 'image 2' work in principle, but the model can attend to the wrong image, especially when the images look similar. Co-locating each image with its question makes the binding **structural rather than referential**, the model does not have to figure out which image goes with which question, because they are adjacent. The pattern that works: ``` [image_1] What type of architecture is shown here? [image_2] What material is the facade made of? [image_3] Approximately what decade is this from? ``` The pattern that fails: ``` [image_1] [image_2] [image_3] For the first image, what type of architecture is shown? For the second image, what material is the facade made of? For the third image, approximately what decade is this from? ``` The second

TL;DR

Put the image immediately before the text that asks about it; co-locating image and question removes reference ambiguity in multi-image prompts and aligns with the recency the model uses for both vision and text tokens.

Memory aid

Easy to grasp

Imagine showing a friend a photo and asking 'what do you see?' If you show the photo, then ask, the connection is obvious. If you ask first and pull out the photo later, your friend has to remember the question while looking at the picture. Now imagine showing three different photos and asking three different questions in a jumbled order, your friend will get confused about which question goes with which photo. The trick is to pair each photo with its question right next to each other. Models work the same way.

Key concepts

Context Engineering

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Image placement is one of those small context-engineering decisions that has measurable quality consequences and that engineers often get wrong by defaulting to the visually clean pattern of stacking images at the top and questions at the bottom. The right rule is the opposite: each image goes immediately before the text that asks about it, and the pair stays adjacent. The rule is universal across 2026 multimodal providers and has two distinct rationales, positional attention bias and reference resolution, that compound when prompts have multiple images. This deep dive walks through the two rationales, the failure mode of clustered layouts, the cross-provider tokenization notes, and the extension to video and audio.

Positional attention bias for vision tokens

The lost-in-the-middle finding (Liu et al. 2023) showed that text content positioned in the middle of a long context is recalled less reliably than the same content at the ends. The U-shaped recall curve has been re-validated repeatedly in subsequent work, and 2024-2025 multimodal studies have confirmed that the effect extends to vision tokens.

An image placed at the start of a long mixed-modal prompt with the question at the end forces the model to attend back across a wall of text to ground its answer. Frontier models in 2026 are better at this than 2023 vision-language models, but the bias has not vanished. The dip is shallower; the principle is the same.

The practical consequence is that the recency of the image at the moment the question is read affects answer quality. Placing the image immediately before the question puts the vision tokens at maximum salience for the attention pattern that will produce the answer. Placing the image far away forces the model to do more work to maintain that grounding.

This is the first-order reason for the placement rule. It applies even to single-image prompts, although the effect is small when prompts are short and the question is unambiguous about which image it refers to.

Reference resolution in multi-image prompts

Provider-specific tokenization in 2026

Extension to video and audio

Practical engineering checklist

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Real products, models, and research that use this idea.

Anthropic's Claude SDK pattern interleaves image and text content blocks for multi-image prompts in production
OpenAI's GPT-5.5 documentation recommends image then question pairing for multi-image visual QA

What an interviewer would ask next. Try answering before peeking at the approach.

QHow would you arrange a prompt that has 20 images and one summarizing question across all of them?

Co-locate each image with a brief per-image question or label that grounds it: 'image_1: what is shown? image_2: what is shown?' and so on. Then place the cross-image summarizing question after all the pairs. This forces the model to ground each image individually before reasoning across them. A pure stack of 20 images followed by 'summarize' typically conflates content from images that look similar.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Putting all images at the top of the prompt and then asking multiple questions about them in a later block, leaving the model to guess which question maps to which image.

60 second bullets to scan on the way to the call.

State the rule: image immediately before the question that references it
Explain the recency rationale (positional attention bias applies to vision tokens too)

Primary sources. Browse if you want the original framing.