Explain why the vision-encoder phase of a multimodal model (ViT + LLM) creates serving headaches when co-located on the same GPU as LLM decode. What does production do to fix it?
Free · 2 AI evals / day
A multimodal model has two very different engines. The **vision encoder** (a ViT) turns an image into a block of tokens. The language model then reads those image tokens plus the text and generates a reply one token at a time.
These two engines stress the GPU in opposite ways. The vision encoder runs lots of big matrix multiplies over a fixed grid of image patches, all at once. That keeps the tensor cores busy, so it is compute-bound. It also finishes quickly, in one short burst of roughly 10 to 50 milliseconds per image.
LLM decode is the opposite. Each decode step produces a single token and spends most of its time reading the KV cache out of memory. It is bandwidth-bound and runs continuously, step after step, for the whole reply.
The trouble starts when both share one GPU. The vision burst grabs nearly all the compute units for its duration. Every chat request that was quietly decoding suddenly stalls, waiting for the burst to clear. Users see latency spikes that have nothing to do with their own request.
Production fixes this by separating the two. Route vision work to its own pool of GPUs, or chop the encoder into chunks and interleave them with decode steps so neither one starves the other.
**The vision encoder and the LLM decoder sit on opposite ends of the roofline.** A ViT is compute-bound. It runs dense matmuls over a fixed image grid with high arithmetic intensity and high data reuse. It executes as one short burst, roughly 10 to 50 milliseconds per image, saturating the tensor cores.
LLM decode is bandwidth-bound. Each step emits one token and is gated by streaming the KV cache from HBM. It is steady and continuous, every step in the 10 to 30 millisecond range across the whole reply.
Co-locate them and they collide. The encoder burst monopolizes the streaming multiprocessors, and during its attention it also eats HBM bandwidth. Every concurrent decode stalls for the burst duration. Time-per output token spikes for all chat users, including ones who sent no image at all.
Production borrows the prefill versus decode playbook. First, dedicated encoder pools route ViT work to compute-optimized GPUs, then ship the image-token features to the decode pool. Second, chunked encoder forward interleaves with decode steps, so the burst never runs uninterrupted. Third, image-token caching lets identical images, like a fixed UI icon set, skip re-encoding entirely.
The lesson generalizes. Different bottleneck regimes on one GPU interfere. Isolate them onto separate hardware, or interleave them so neither starves the other.
Quick reference
TL;DR
A ViT encoder is a compute-bound burst; LLM decode is bandwidth-bound and steady. Co-located, the burst monopolizes the GPU and stalls every in-flight decode, so production isolates or interleaves them.
Picture one kitchen serving two kinds of orders. The vision encoder is a giant catering job: it briefly grabs every burner and oven at once, runs hot for a few seconds, then finishes. The chat decode is a steady drip of tiny single plates, each one waiting on the pantry rather than the stove. While the catering job hogs all the burners, every small plate just sits there getting cold, even though they barely needed a burner at all. Customers waiting on their tiny plates suddenly see long, jittery delays. The fix is the same one any busy kitchen reaches for: give the catering job its own dedicated stoves, or force it to cook in small chunks between the little plates so nobody starves.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Text-only workloads with no image inputs at all There is no vision encoder running, so there is no burst to isolate and no interference to manage.
Tiny deployment serving one request at a time With no concurrent decode to stall, the encoder burst harms only its own request and isolation adds needless complexity.
Offline batch image processing where latency does not matter Tail latency is irrelevant, so you should maximize encoder throughput rather than protect interactive decode.
Image volume so low that a dedicated encoder pool sits idle A separate pool wastes hardware, so interleaving or simple co-location is cheaper until image traffic grows.
Quick scan · 2–4 min
Everything important, quickly.
4 min: two rooflines (ViT compute-bound burst vs decode bandwidth-bound) + the SM monopoly and tail latency spike + three fixes (encoder pool disaggregation, chunked interleaving, image-token caching).
Real-world examples
Real products, models, and research that use this idea.
1vLLM and SGLang both serve multimodal models like Llama 4 and Qwen 3 VL, where the ViT encoder forward competes with decode on the same device unless scheduled apart.
2DistServe popularized prefill-decode disaggregation, and the same routing pattern is reused to place vision encoders on dedicated compute-optimized GPU pools.
3Gemini 3.1 Pro and GPT-5.5 vision endpoints handle high image-token counts per request, making encoder bursts a first-order latency concern at production scale.
4Claude Opus 4.7 multimodal serving caches encoded features for repeated images, so shared UI screenshots or icon sets skip re-encoding on subsequent requests.
5NVIDIA TensorRT-LLM exposes chunked and interleaved scheduling primitives used to keep a heavy encoder forward from monopolizing SMs during concurrent decode.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy does an encoder burst hurt text-only requests that sent no image?
Sign in or go Pro to see the approach for this follow-up.
QHow does disaggregating the encoder change the data you ship across the interconnect?
Sign in or go Pro to see the approach for this follow-up.
QWhen does image-token caching actually pay off, and how do you key the cache?
Sign in or go Pro to see the approach for this follow-up.
QHow does image-token count interact with KV cache pressure on the decode pool?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Treating the vision encoder as just more prefill. It is a separate compute-bound burst with its own roofline, and ignoring that interference is why co-located decode latency spikes.