Keeping an image-heavy multimodal feature feeling fast despite a slow vision pipeline

Attempt it

A product lets users snap a photo and ask a question about it, but the round trip feels sluggish because of image upload, encoding, and a large vision-token prefill. Describe how you would keep the experience acceptable without simply maxing out the image pipeline.

Free · 2 AI evals / day

The round trip feels slow because of three stages: uploading the image, encoding it into vision tokens, and the large prefill the model has to process before it can answer. Some of this you can genuinely cut, and some you can only make feel shorter. The biggest real win is the **image-token budget**. A high-resolution image gets tiled into many patches, and each patch costs tokens. The prefill grows with resolution. So instead of always sending the highest resolution, match the resolution to the task. A receipt or a road sign does not need 4K tiling; a small image answers the question and the prefill shrinks dramatically. The rest is about perceived latency. Stream the text answer token by token so the user watches it appear instead of staring at a frozen spinner. Show an explicit upload and processing state so the wait feels intentional rather than broken. Where it fits, return a fast first-pass result — a quick low-resolution read or a cheap caption — and refine if the user needs more. Finally, cache the encoded image tokens so follow-up questions on the same photo do not re-encode it. The goal is to spend tokens where accuracy needs them and make the unavoidable latency feel responsive.

Split the problem into latency you can cut and latency you can only mask, because the image round trip has both. The cuttable part is the **vision-token prefill**, which dominates time to first token for image inputs. Prefill cost grows with resolution because high-res images are tiled into many patches, and each patch becomes tokens the model must process before generating. The first lever is to set a resolution and token budget matched to the use case rather than always maxing resolution. A receipt, a screenshot, or a road sign answers fine at modest resolution. Reserving 4K tiling for tasks that genuinely need fine detail keeps the prefill small for the common case. The maskable part is everything the user perceives. Stream the text answer token by token so they watch progress instead of a frozen spinner — this changes the felt latency even when total time is unchanged. Show an explicit upload and encoding state so each stage of the wait reads as intentional. Where the task allows, return a fast first-pass result — a low-resolution read or a cheap caption — and refine on demand, rather than blocking on a maxed-out pipeline. Finally, **cache encoded image tokens** so follow-up questions about the same image skip re-encoding entirely; only the new text prompt is added. The throughline is to spend image tokens where accuracy demands it and make the unavoidable latency feel responsive everywhere else.

Quick reference

TL;DR

Cut real latency by matching resolution and image-token budget to the task, then mask the rest with streaming, an explicit processing state, a fast first-pass result, and cached image tokens.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Imagine ordering a coffee. The barista can't make it instantly, but the wait feels fine because you see them grinding, pouring, and steaming — there's progress the whole time. Now imagine they vanish into a back room and reappear five minutes later with the cup. Same wait, but it felt awful because nothing was happening in front of you. A photo question feature is the same. Some of the wait is unavoidable: uploading the picture and having the model read it takes time. So you do two things. First, don't send a giant 4K photo when a small one would answer a question about a receipt — that's like grinding way more beans than the cup needs. Second, show the user every step and stream the answer word by word, so the wait feels like a barista working, not a frozen screen.

Key concepts

Multimodal AI LLM System Design

Detailed answer & concept explanation

~6 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Sign in for 5 free unlocks/day or go Pro for unlimited. See Pro pricing.

Quick scan · 2–4 min

Everything important, quickly.

Open by decomposing the round trip into upload, encode, and prefill, and identify the vision-token prefill as the dominant, cuttable cost that scales with resolution and tiling. Make the case for a task-matched resolution budget over always sending max resolution. Then pivot to perceived latency: streaming the answer, an explicit per-stage progress indicator, and a fast first-pass result that refines on demand. Close with the multi-turn efficiency win of caching encoded image tokens, and the throughline — spend tokens where accuracy needs them, make the rest feel responsive.

Real-world examples

Real products, models, and research that use this idea.

GPT-5.5 and Gemini 3.1 Pro bill image inputs by tiles, so a high-resolution photo costs many times the tokens of a thumbnail — the same lever that drives prefill latency
Claude Opus 4.7 and other VLMs stream the text answer token by token, which is what makes a multi-second prefill feel responsive
Receipt and document scanning apps downscale before sending because a small image reads the totals fine and slashes the prefill
Multi-turn image chat (ask several questions about one uploaded photo) benefits from caching the encoded image so only the new prompt is processed
Computer-use agents reading a screen cap resolution to keep per-step prefill low, since they take many steps over the same kind of image

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QHow would you decide the resolution budget per request automatically instead of one global setting?

Sign in or go Pro to see the approach for this follow-up.

QWhat breaks when you cache encoded image tokens across turns, and how do you keep it correct?

Sign in or go Pro to see the approach for this follow-up.

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Always sending the highest-resolution image to be safe, which blows up the vision-token prefill and latency for tasks that a small, cheap image would have answered just fine.

Sign in or go Pro to see all red flags and common mistakes. See Pro pricing.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.