Explain why the vision encoder phase of a multimodal model (ViT + LLM) creates serving headaches when co-located on the same GPU as LLM decode. What does production do to fix it?
A ViT encoder is a compute-bound burst; LLM decode is bandwidth-bound and steady. Co-located, the burst monopolizes the GPU and stalls every in-flight decode, so production isolates or interleaves them.
Picture one kitchen serving two kinds of orders. The vision encoder is a giant catering job: it briefly grabs every burner and oven at once, runs hot for a few seconds, then finishes. The chat decode is a steady drip of tiny single plates, each one waiting on the pantry rather than the stove. While the catering job hogs all the burners, every small plate just sits there getting cold, even though they barely needed a burner at all. Customers waiting on their tiny plates suddenly see long, jittery delays. The fix is the same one any busy kitchen reaches for: give the catering job its own dedicated stoves, or force it to cook in small chunks between the little plates so nobody starves.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Multimodal serving breaks a comfortable assumption that single-modality LLM engineers carry around: that a request has exactly two phases, prefill and decode, and that you optimize the boundary between them. Add a vision tower and a third regime appears. The vision encoder is neither prefill nor decode. It is a dense, compute-bound burst with its own roofline, its own latency profile, and its own scheduling needs.
The practical consequence is interference. When a ViT encoder and an LLM decoder share one GPU, they pull the hardware toward opposite ceilings at the same time. The encoder wants every tensor core for a short, intense window. Decode wants steady memory bandwidth across a long sequence of tiny steps. Whoever holds the streaming multiprocessors during the burst wins, and decode loses, producing latency spikes that confuse anyone who only measured average throughput.
This deep dive builds the roofline picture, quantifies the burst, explains why even text-only users get hurt, and then walks through the three production responses: encoder disaggregation, chunked interleaving, and image-token caching. By the end you should be able to defend a multimodal serving design under mixed traffic and explain exactly which latency metric each fix protects.
Two engines, two rooflines
The roofline model classifies a kernel by arithmetic intensity, the ratio of FLOPs to bytes moved. High intensity means the kernel is compute-bound and lives near the FLOPs ceiling. Low intensity means it is memory-bound and lives near the bandwidth ceiling. Where a kernel lands relative to the ridge point tells you which hardware resource you will run out of first.
A ViT encoder sits firmly on the compute side. It runs dense matmuls over a fixed grid of image patches, with heavy weight reuse across patches and across the batch. Patch projection, the attention blocks, and the MLP blocks are all large, regular matrix multiplies that keep the tensor cores saturated. Because every patch reuses the same weight matrices, the bytes moved per FLOP stay low, which pins the kernel high on the roofline. Arithmetic intensity is high, so the encoder is bound by raw compute, not memory.
LLM decode sits on the opposite side. Each decode step processes a single new token, multiplies relatively small activations against weights, and spends most of its time streaming the KV cache from HBM. The matrices are tall and skinny, weights are read once and used once per step, and there is almost no reuse to amortize the memory traffic. Arithmetic intensity is low, so the bottleneck is memory bandwidth. The two engines literally optimize against different ceilings, which is why one device cannot keep both happy at once. Tuning the GPU to feed the encoder's tensor cores does nothing for decode, and widening memory bandwidth for decode leaves the encoder's compute ceiling untouched.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- vLLM and SGLang both serve multimodal models like Llama 4 and Qwen 3 VL, where the ViT encoder forward competes with decode on the same device unless scheduled apart.
- DistServe popularized prefill-decode disaggregation, and the same routing pattern is reused to place vision encoders on dedicated compute-optimized GPU pools.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy does an encoder burst hurt text-only requests that sent no image?
Trace what physical resource is shared. Decode needs SM cycles and HBM bandwidth even though it is bandwidth-bound. The encoder pins both for its burst duration, so any decode on the same device stalls regardless of whether that request involves an image.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Treating the vision encoder as just more prefill. It is a separate compute-bound burst with its own roofline, and ignoring that interference is why co-located decode latency spikes.
60 second bullets to scan on the way to the call.
Why a ViT encoder is compute-bound while LLM decode is bandwidth-bound
Typical encoder burst duration and decode per-step latency ranges
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.