Why is the vision-encoder phase a different bottleneck class from LLM decode in a multi-modal model?

A multimodal model has two very different engines. The **vision encoder** (a ViT) turns an image into a block of tokens. The language model then reads those image tokens plus the text and generates a reply one token at a time. These two engines stress the GPU in opposite ways. The vision encoder runs lots of big matrix multiplies over a fixed grid of image patches, all at once. That keeps the tensor cores busy, so it is compute-bound. It also finishes quickly, in one short burst of roughly 10 to 50 milliseconds per image. LLM decode is the opposite. Each decode step produces a single token and spends most of its time reading the KV cache out of memory. It is bandwidth-bound and runs continuously, step after step, for the whole reply. The trouble starts when both share one GPU. The vision burst grabs nearly all the compute units for its duration. Every chat request that was quietly decoding suddenly stalls, waiting for the burst to clear. Users see latency spikes that have nothing to do with their own request. Production fixes this by separating the two. Route vision work to its own pool of GPUs, or chop the encoder into chunks and interleave them with decode steps so neither one starves the other.

**The vision encoder and the LLM decoder sit on opposite ends of the roofline.** A ViT is compute-bound. It runs dense matmuls over a fixed image grid with high arithmetic intensity and high data reuse. It executes as one short burst, roughly 10 to 50 milliseconds per image, saturating the tensor cores. LLM decode is bandwidth-bound. Each step emits one token and is gated by streaming the KV cache from HBM. It is steady and continuous, every step in the 10 to 30 millisecond range across the whole reply. Co-locate them and they collide. The encoder burst monopolizes the streaming multiprocessors, and during its attention it also eats HBM bandwidth. Every concurrent decode stalls for the burst duration. Time-per output token spikes for all chat users, including ones who sent no image at all. Production borrows the prefill versus decode playbook. First, dedicated encoder pools route ViT work to compute-optimized GPUs, then ship the image-token features to the decode pool. Second, chunked encoder forward interleaves with decode steps, so the burst never runs uninterrupted. Third, image-token caching lets identical images, like a fixed UI icon set, skip re-encoding entirely. The lesson generalizes. Different bottleneck regimes on one GPU interfere. Isolate them onto separate hardware, or interleave them so neither starves the other.