Cutting the number of vision tokens a VLM sends to the LLM without wrecking accuracy

Short answer·Medium·4.0 · 0·~3 min

Attempt it

Image-heavy serving is dominated by vision-token prefill. Explain the main techniques for reducing how many vision tokens reach the LLM, and the tradeoff each one makes.

Free · 2 AI evals / day

When a VLM serves images, most of the cost is the **vision tokens**. The encoder splits each image into patches and turns them into a big set of patch tokens — often hundreds per image — and the LLM has to process all of them. So the way to save money is to send fewer of those tokens to the LLM. There are three main ways to shrink the set, in rising order of cleverness. **Pooling** averages or strides over neighboring patch tokens to make a smaller grid. It is dead simple and fast, but blunt: it throws away fine detail evenly, including small text. **Token merging** is smarter. It combines tokens that are similar to each other, so flat regions like a plain wall collapse while busy regions keep their tokens. You lose less useful signal for the same token count. A **learned resampler**, like a Q-Former, uses a small fixed set of query tokens that look over all the patches and summarize them into a constant small number. It bounds the cost tightly but needs extra training. Every method trades fine detail for speed and cost. The right amount of compression depends on whether the task needs to read small text or just get the gist.

Vision-token prefill dominates image-heavy serving because the encoder emits many patch tokens per image and the LLM must attend over every one. The lever is to compress that patch-token set in the connector, **before** it reaches the LLM. Three techniques cover the field. **Pooling** strides or averages over neighboring patch tokens to shrink the grid. It is cheap and trivial to implement, but blunt, since it discards fine spatial detail uniformly whether or not a region mattered. Token merging is adaptive. It combines tokens that are similar and leaves distinct regions intact, so a plain background collapses while a dense chart keeps its tokens. For the same final count it preserves more accuracy than pooling, at modest extra complexity. A learned resampler — Q-Former or perceiver-style — uses a small fixed set of query tokens that cross-attend over all patch tokens. They emit a constant, small output count regardless of input size. That pins cost tightly and decouples it from resolution, but it adds parameters and has to be trained. The shared tradeoff is detail for cost. All three sacrifice the same things first — small text, dense charts, exact object counts — in exchange for large latency and cost savings. So the design choice is task-driven: keep more tokens when the job needs dense reading, compress hard when the job only needs the overall gist.

Quick reference

TL;DR

Shrink the patch-token set before the LLM: pooling (cheap, blunt), token merging (adaptive, keeps busy regions), or a learned resampler (fixed small count). All trade fine detail for latency and cost.

Memory aid

Easy to grasp

Imagine summarizing a detailed painting for someone over the phone. You cannot describe every brushstroke, so you compress. The crude way is to split the painting into a grid and give one average color per cell — fast, but you lose the tiny signature in the corner. A smarter way is to spend more words on the busy, interesting areas and barely mention the plain sky. A third way is to decide in advance you will use exactly twenty sentences no matter how complex the painting, and pick the twenty most useful things to say. Each method gets the gist across cheaply, but all of them risk dropping the small print — the fine details only a full description would carry.

Key concepts

Multimodal AI Inference Optimization Attention

Detailed answer & concept explanation

~6 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Quick scan · 2–4 min

Everything important, quickly.

Spend about 5 minutes: establish that vision tokens are the prefill cost and compression sits in the connector, then walk the three techniques in rising fidelity, ending on the task-driven tradeoff.

Real-world examples

Real products, models, and research that use this idea.

Q-Former in BLIP-2 — a learned resampler that compresses any image's patch tokens to a small fixed set of query tokens before the LLM.
Token merging (ToMe) — adaptively merges similar ViT tokens to cut sequence length while preserving accuracy on busy regions.
LLaVA-style pixel-shuffle or pooling — strides over patch tokens to shrink the grid cheaply when fine reading is not required.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QYour VLM does great on natural photos but fails on document OCR after enabling compression. What is happening and how do you fix it?

QHow does a learned resampler's fixed token count interact with high-resolution tiling, where one image becomes many tiles?

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Thinking compression happens inside the LLM. It happens on the patch tokens between the vision encoder and the LLM — shrink the set there, before prefill, or you save nothing.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

References

Primary sources. Skim if you want the original framing.