Explain how a resampler connector bounds vision-token cost, and what it gives up to do so

Start with the plain **MLP projector**. It takes each patch feature from the vision encoder and maps it, one to one, into a token the language model reads. So if a high-resolution image tiles into 2,000 patches, the LLM gets 2,000 vision tokens. More resolution means more patches, which means more tokens and more cost on every attention layer. A **resampler** — a Q-Former or perceiver resampler — replaces that mapping. It holds a small fixed set of learnable query tokens, maybe 32 or 64. Those queries cross-attend over all the patch features and emit exactly that many output tokens, no matter how many patches arrived. The result is that the LLM's image cost is decided by the query count, not by resolution. That makes latency and memory predictable. The tradeoff is real: cramming a dense image into 32 tokens loses fine detail, so reading small text, counting objects, or judging exact layout all get worse. The query count is a budget you tune to the task.

The MLP projector and the resampler differ in one structural fact: what sets the vision-token count handed to the LLM. An **MLP projector** maps patches one to one, so the count equals the patch count. Patch count scales with resolution and tiling, and every vision token lands in the LLM's sequence, where attention cost grows with length. So high-res tiling directly inflates sequence length, latency, and memory. A resampler breaks that coupling. It introduces a small fixed set of learnable query tokens — parameters of the connector, not image embeddings. These queries are the queries in a cross-attention step; the patch features supply keys and values. Each query attends over the whole image and writes one output token, so the number of outputs equals the number of queries and nothing else. Send 256 patches or 4,000 patches, you still get 32 tokens out. Now the LLM's per-image cost is a constant you choose at design time, independent of resolution. The price is an information bottleneck. A fixed query budget cannot carry every fine detail of a dense, high-resolution page, so dense-text reading, small objects, and exact counts degrade relative to forwarding all patches. The query count is the knob: more queries recover detail at higher cost. You size it to the hardest task you must support, not to either extreme.