Keeping an image heavy multimodal feature feeling fast despite a slow vision pipeline

Short answer·Medium·4.0 · 0·~3 min·Asked atDoordash Redis Swiggy

Attempt it

A product lets users snap a photo and ask a question about it, but the round trip feels sluggish because of image upload, encoding, and a large vision token prefill. Describe how you would keep the experience acceptable without simply maxing out the image pipeline.

Free · 2 AI evals / day

The round trip feels slow because of three stages: uploading the image, encoding it into vision tokens, and the large prefill the model has to process before it can answer. Some of this you can genuinely cut, and some you can only make feel shorter. The biggest real win is the **image-token budget**. A high-resolution image gets tiled into many patches, and each patch costs tokens. The prefill grows with resolution. So instead of always sending the highest resolution, match the resolution to the task. A receipt or a road sign does not need 4K tiling; a small image answers the question and the prefill shrinks dramatically. The rest is about perceived latency. Stream the text answer token by token so the user watches it appear instead of staring at a frozen spinner. Show an explicit upload and processing state so the wait feels intentional rather than broken. Where it fits, return a fast first-pass result — a quick low-resolution read or a cheap caption — and refine if the user needs more. Finally, cache the encoded image tokens so follow-up questions on the same photo do not re-encode it. The goal is to spend tokens where accuracy needs them and make the unavoidable latency feel responsive.

Split the problem into latency you can cut and latency you can only mask, because the image round trip has both. The cuttable part is the **vision-token prefill**, which dominates time to first token for image inputs. Prefill cost grows with resolution because high-res images are tiled into many patches, and each patch becomes tokens the model must process before generating. The first lever is to set a resolution and token budget matched to the use case rather than always maxing resolution. A receipt, a screenshot, or a road sign answers fine at modest resolution. Reserving 4K tiling for tasks that genuinely need fine detail keeps the prefill small for the common case. The maskable part is everything the user perceives. Stream the text answer token by token so they watch progress instead of a frozen spinner — this changes the felt latency even when total time is unchanged. Show an explicit upload and encoding state so each stage of the wait reads as intentional. Where the task allows, return a fast first-pass result — a low-resolution read or a cheap caption — and refine on demand, rather than blocking on a maxed-out pipeline. Finally, **cache encoded image tokens** so follow-up questions about the same image skip re-encoding entirely; only the new text prompt is added. The throughline is to spend image tokens where accuracy demands it and make the unavoidable latency feel responsive everywhere else.

TL;DR

Cut real latency by matching resolution and image-token budget to the task, then mask the rest with streaming, an explicit processing state, a fast first-pass result, and cached image tokens.

Memory aid

Easy to grasp

Imagine ordering a coffee. The barista can't make it instantly, but the wait feels fine because you see them grinding, pouring, and steaming — there's progress the whole time. Now imagine they vanish into a back room and reappear five minutes later with the cup. Same wait, but it felt awful because nothing was happening in front of you. A photo question feature is the same. Some of the wait is unavoidable: uploading the picture and having the model read it takes time. So you do two things. First, don't send a giant 4K photo when a small one would answer a question about a receipt — that's like grinding way more beans than the cup needs. Second, show the user every step and stream the answer word by word, so the wait feels like a barista working, not a frozen screen.

Key concepts

Multimodal AI LLM System Design

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

When a photo-question feature feels sluggish, the instinct is to throw hardware at it — a bigger GPU, a faster model, more replicas. That instinct misreads the problem. The latency in a multimodal round trip has a specific shape, and most of it is not a hardware shortage; it is a token-budget decision and a perception-design decision wearing a hardware costume.

This deep dive separates the two kinds of latency that live in an image round trip: the part you can genuinely cut by spending fewer vision tokens, and the part you can only make feel shorter through interface design. We will walk through where the time actually goes, why the prefill is the lever, how a task-matched resolution budget changes the economics, how streaming and staged progress reshape the felt wait, and how caching the encode rescues multi-turn image chat.

Where the seconds actually go

Decompose the round trip into three stages and the diagnosis becomes obvious.

Upload moves the image bytes over the network. Its cost scales with file size and the user's connection, and for a typical phone photo it is real but bounded — and it is the stage you have the least control over.

Encode runs the image through the vision encoder, splitting it into patches and producing vision tokens. Prefill is the model attending over those vision tokens plus the text prompt before it can emit the first output token.

For image inputs, prefill is usually the dominant contributor to time to first token, and the reason is the token count. A high-resolution image tiled at fine granularity becomes hundreds to thousands of vision tokens, and prefill compute scales with that count. The key realization is that the slow part is not a slow network or a slow GPU — it is that you handed the model a huge prefill. That reframes the whole problem from 'buy more compute' to 'send fewer tokens where you can, and mask the rest.'

The resolution budget: the one real cut

Streaming and staging: masking what you can't cut

Fast first-pass: give them something to act on

Caching the encode for multi-turn image chat

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Real products, models, and research that use this idea.

GPT-5.5 and Gemini 3.1 Pro bill image inputs by tiles, so a high-resolution photo costs many times the tokens of a thumbnail — the same lever that drives prefill latency
Claude Opus 4.7 and other VLMs stream the text answer token by token, which is what makes a multi-second prefill feel responsive

What an interviewer would ask next. Try answering before peeking at the approach.

QHow would you decide the resolution budget per request automatically instead of one global setting?

Route on the task. A cheap classifier or the user's intent can flag whether fine detail is needed — dense table, small text, medical scan — versus a coarse scene question. Default to a low tier and escalate tiling only when the signal says detail matters. You can also do a cheap low-res pass first and re-encode at higher resolution only if the model expresses uncertainty.

1 more follow-up an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Always sending the highest-resolution image to be safe, which blows up the vision-token prefill and latency for tasks that a small, cheap image would have answered just fine.

60 second bullets to scan on the way to the call.

Decompose the round trip into upload, encode, and prefill stages
Explain why prefill scales with resolution and tiling and dominates time to first token

Primary sources. Browse if you want the original framing.