Diagnose why a projection-style VLM hits its context limit on 20-image requests

Spot the error·Hard·4.0 · 0·~2 min

Attempt it

Click any words you think contain an error. Click again to unmark.

Mark at least one word to submit.

The flawed claim is that the projection adapter adds no tokens to the sequence, so image count cannot be filling the context. That is backwards. A **projection-adapter** connector exists to put visual features into the LLM's token sequence. It runs the encoder's output through a small MLP that maps each image's features into the language model's token space, and those projected vectors become extra tokens the model has to process. So images absolutely consume sequence length. The overflow scales with image count for exactly that reason. Each image becomes many tokens — often hundreds, and far more for high-resolution images that get split into tiles. Twenty high-res images can become tens of thousands of tokens before any text is added, which is what overflows the window. The fix is to stop spending sequence length per image. One option is a cross-attention connector, where the LLM keeps its text tokens and attends to visual features held outside the sequence, so image count no longer inflates context. Another is a resampler that compresses each image down to a small fixed number of query tokens before they enter the sequence. Either way, you break the link between how many images you send and how full the context gets.

The sentence reverses how a **projection-adapter** works. The connector's entire purpose is to map encoder features into the LLM's token embedding space, so each image becomes extra tokens inside the sequence. Far from adding no tokens, it is the reason images consume context at all. That is why the overflow scales with image count. One image projects to a few hundred tokens, and high-resolution handling via tiling pushes that into the low thousands. So 20 images contribute on the order of tens of thousands of tokens. The vision tokens dominate the sequence and blow the window before the text prompt is even counted. So image count is precisely what fills the context — the opposite of the claim. The correct mental model: in a projection design, sequence length scales with the number and resolution of images. The fix is to decouple those. A **cross-attention** connector keeps the LLM's text tokens and lets them attend to visual features that live outside the sequence, so adding images does not grow the context. A resampler (a Q-Former or Perceiver-style module) compresses each image to a small fixed set of query tokens before projection, capping tokens per image regardless of resolution. Either change breaks the link between image count and context length, which is the property the workload actually needs.

Quick reference

TL;DR

A projection-adapter turns every image into many tokens inside the LLM sequence, so 20 high-res images multiply the context — image count is exactly what fills the window. Fix it with cross-attention or a resampler.

Memory aid

Easy to grasp

Imagine your notebook has a fixed number of pages. Someone claims that taping in 20 photos cannot fill it because the photos go on the wall, not the pages. But this notebook actually transcribes each photo into a long block of writing on the pages themselves. Tape in 20 detailed photos and the writing fills every page before you even start your notes. The mistake is believing the photos stayed off the pages — they did not. To fix it, you either keep the photos on the wall and just glance at them, or you write a tiny one-line summary of each photo instead of the full transcription.

Key concepts

Multimodal AI Transformer Architecture

Detailed answer & concept explanation

~7 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Quick scan · 2–4 min

Everything important, quickly.

Name the error precisely: a projection-adapter inserts visual features as tokens in the sequence, so its no-tokens premise is false. Quantify the overflow as image count times tokens per image, and show how tiling multiplies tokens per image into the thousands. Then give the fix — cross-attention or a resampler to decouple image count from sequence length — and explain why simply enlarging the window only treats the symptom.

Real-world examples

Real products, models, and research that use this idea.

LLaVA-style projection connectors are exactly the design where many or high-res images inflate the sequence and overflow context
AnyRes and dynamic-resolution tiling multiply tokens per image, the common trigger for context blowups on document scans
Flamingo's cross-attention connector is the canonical fix that decouples sequence length from image count
BLIP-2's Q-Former resampler compresses each image to a small fixed query-token set, capping the per-image token budget
Production VLMs handling multi-image requests in 2026 pair a resampler with projection to bound the visual token load

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QIf you must keep the projection-adapter for training simplicity, how do you bound the token blowup?

QHow would you decide between cross-attention and a resampler when context overflow is the symptom?

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Believing a projection-adapter keeps visual features out of the sequence, when its whole job is to insert them as tokens — which is precisely why image count drives the context overflow.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

References

Primary sources. Skim if you want the original framing.