Cutting the number of vision tokens a VLM sends to the LLM without wrecking accuracy

Short answer·Medium·4.0 · 0·~3 min·Asked atMphasis Pinecone Razorpay

Attempt it

Image heavy serving is dominated by vision token prefill. Explain the main techniques for reducing how many vision tokens reach the LLM, and the tradeoff each one makes.

Free · 2 AI evals / day

When a VLM serves images, most of the cost is the **vision tokens**. The encoder splits each image into patches and turns them into a big set of patch tokens — often hundreds per image — and the LLM has to process all of them. So the way to save money is to send fewer of those tokens to the LLM. There are three main ways to shrink the set, in rising order of cleverness. **Pooling** averages or strides over neighboring patch tokens to make a smaller grid. It is dead simple and fast, but blunt: it throws away fine detail evenly, including small text. **Token merging** is smarter. It combines tokens that are similar to each other, so flat regions like a plain wall collapse while busy regions keep their tokens. You lose less useful signal for the same token count. A **learned resampler**, like a Q-Former, uses a small fixed set of query tokens that look over all the patches and summarize them into a constant small number. It bounds the cost tightly but needs extra training. Every method trades fine detail for speed and cost. The right amount of compression depends on whether the task needs to read small text or just get the gist.

Vision-token prefill dominates image-heavy serving because the encoder emits many patch tokens per image and the LLM must attend over every one. The lever is to compress that patch-token set in the connector, **before** it reaches the LLM. Three techniques cover the field. **Pooling** strides or averages over neighboring patch tokens to shrink the grid. It is cheap and trivial to implement, but blunt, since it discards fine spatial detail uniformly whether or not a region mattered. Token merging is adaptive. It combines tokens that are similar and leaves distinct regions intact, so a plain background collapses while a dense chart keeps its tokens. For the same final count it preserves more accuracy than pooling, at modest extra complexity. A learned resampler — Q-Former or perceiver-style — uses a small fixed set of query tokens that cross-attend over all patch tokens. They emit a constant, small output count regardless of input size. That pins cost tightly and decouples it from resolution, but it adds parameters and has to be trained. The shared tradeoff is detail for cost. All three sacrifice the same things first — small text, dense charts, exact object counts — in exchange for large latency and cost savings. So the design choice is task-driven: keep more tokens when the job needs dense reading, compress hard when the job only needs the overall gist.

TL;DR

Shrink the patch-token set before the LLM: pooling (cheap, blunt), token merging (adaptive, keeps busy regions), or a learned resampler (fixed small count). All trade fine detail for latency and cost.

Memory aid

Easy to grasp

Imagine summarizing a detailed painting for someone over the phone. You cannot describe every brushstroke, so you compress. The crude way is to split the painting into a grid and give one average color per cell — fast, but you lose the tiny signature in the corner. A smarter way is to spend more words on the busy, interesting areas and barely mention the plain sky. A third way is to decide in advance you will use exactly twenty sentences no matter how complex the painting, and pick the twenty most useful things to say. Each method gets the gist across cheaply, but all of them risk dropping the small print — the fine details only a full description would carry.

Key concepts

Multimodal AI Inference Optimization Attention

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Ask any team running a vision-language model at scale where the bill comes from and the answer is vision tokens. A single image can turn into hundreds or thousands of patch tokens, and the LLM has to run prefill attention over every one before it generates a word. Stack a few images per request and the visual tokens dwarf the text.

That is why vision-token compression is one of the highest-leverage knobs in a production VLM. Cutting the patch-token set in half roughly halves prefill cost and latency, which at scale is real money. The catch is that those tokens carry the image's detail, so naive compression quietly destroys the model's ability to read small text or count objects.

This deep dive establishes where compression happens and why it is possible, then walks the three techniques from blunt to clever — pooling, token merging, learned resampler — and closes on the tradeoff that ties them together, with concrete token numbers so the savings and the losses are both visible.

Why vision tokens are the bottleneck and why they compress

A VLM encodes an image by splitting it into a grid of patches and running each through a vision transformer, producing one token per patch. A 336-pixel image at a 14-pixel patch size yields a 24×24 grid — about 576 patch tokens for one ordinary image, and high-resolution tiling multiplies that several-fold.

The LLM then processes all of those tokens during prefill, and attention cost grows with sequence length. So a handful of images can cost more than a long text prompt, and prefill — not generation — becomes the dominant latency.

The reason this is fixable is redundancy. Adjacent patches of a blank wall, a clear sky, or a document margin produce nearly identical features. The patch grid carries far less information than its token count suggests. Compression exploits that: it removes or fuses redundant tokens so the LLM processes a smaller, denser set. Crucially this happens in the connector, the bridge between the vision encoder and the LLM, before any tokens enter prefill. Compress there and every downstream cost shrinks; try to do it inside the LLM and you have already paid for the tokens.

Pooling: the blunt, free downsample

Token merging: spend the budget where the image is busy

Learned resamplers: a constant token count by design

Choosing a method, and how compression stacks with resolution

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Real products, models, and research that use this idea.

Q-Former in BLIP-2 — a learned resampler that compresses any image's patch tokens to a small fixed set of query tokens before the LLM.
Token merging (ToMe) — adaptively merges similar ViT tokens to cut sequence length while preserving accuracy on busy regions.

What an interviewer would ask next. Try answering before peeking at the approach.

QYour VLM does great on natural photos but fails on document OCR after enabling compression. What is happening and how do you fix it?

Tie OCR failure to small-text tokens being merged or pooled away; propose a higher token budget or resolution-aware policy for document inputs specifically.

1 more follow-up an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Thinking compression happens inside the LLM. It happens on the patch tokens between the vision encoder and the LLM — shrink the set there, before prefill, or you save nothing.

60 second bullets to scan on the way to the call.

Why vision-token prefill dominates image-heavy serving cost
Where compression happens: the connector, before the LLM

Primary sources. Browse if you want the original framing.