Keeping an image-heavy multimodal feature feeling fast despite a slow vision pipeline

The round trip feels slow because of three stages: uploading the image, encoding it into vision tokens, and the large prefill the model has to process before it can answer. Some of this you can genuinely cut, and some you can only make feel shorter. The biggest real win is the **image-token budget**. A high-resolution image gets tiled into many patches, and each patch costs tokens. The prefill grows with resolution. So instead of always sending the highest resolution, match the resolution to the task. A receipt or a road sign does not need 4K tiling; a small image answers the question and the prefill shrinks dramatically. The rest is about perceived latency. Stream the text answer token by token so the user watches it appear instead of staring at a frozen spinner. Show an explicit upload and processing state so the wait feels intentional rather than broken. Where it fits, return a fast first-pass result — a quick low-resolution read or a cheap caption — and refine if the user needs more. Finally, cache the encoded image tokens so follow-up questions on the same photo do not re-encode it. The goal is to spend tokens where accuracy needs them and make the unavoidable latency feel responsive.

Split the problem into latency you can cut and latency you can only mask, because the image round trip has both. The cuttable part is the **vision-token prefill**, which dominates time to first token for image inputs. Prefill cost grows with resolution because high-res images are tiled into many patches, and each patch becomes tokens the model must process before generating. The first lever is to set a resolution and token budget matched to the use case rather than always maxing resolution. A receipt, a screenshot, or a road sign answers fine at modest resolution. Reserving 4K tiling for tasks that genuinely need fine detail keeps the prefill small for the common case. The maskable part is everything the user perceives. Stream the text answer token by token so they watch progress instead of a frozen spinner — this changes the felt latency even when total time is unchanged. Show an explicit upload and encoding state so each stage of the wait reads as intentional. Where the task allows, return a fast first-pass result — a low-resolution read or a cheap caption — and refine on demand, rather than blocking on a maxed-out pipeline. Finally, **cache encoded image tokens** so follow-up questions about the same image skip re-encoding entirely; only the new text prompt is added. The throughline is to spend image tokens where accuracy demands it and make the unavoidable latency feel responsive everywhere else.