A Q-Former or perceiver resampler uses a small fixed set of learnable query tokens to cross-attend over many patch features and emit a constant token count — bounding the LLM's image cost.
Imagine you photographed every page of a thick book and want a friend to discuss it, but they will only read a single index card. You hand a few interns a fixed set of blank cards and tell them to skim every page and write down what matters. No matter how many pages there were, you always get back the same small stack of cards. That is what a Q-Former does inside a vision model. The pages are the image patches, the interns are the query slots, and the friend is the language model that only ever sees those few cards.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Every vision-language model has to answer one plumbing question: how do the outputs of a vision encoder actually enter a language model that only understands tokens? The vision side produces a pile of patch features whose size depends on the image, and that variability is awkward for a transformer whose cost grows with sequence length.
The Q-Former and the perceiver resampler are two names for the same idea — a connector that compresses that variable pile into a fixed, small set of tokens before it reaches the LLM. Interviewers reach for this because it sits right at the intersection of two things they care about: how multimodal fusion works, and how you keep image inference affordable.
This deep dive builds the mechanism from the ground up: why a variable patch count is a problem, how a fixed set of learnable queries absorbs the patches via cross-attention, why the output is constant no matter the input, and what that compression costs you on the tasks that need fine detail. The goal is for the constant-output property to feel inevitable rather than magical by the end.
Why a variable bag of patch features is awkward
A vision encoder splits an image into patches and emits one feature vector per patch. A small thumbnail might yield a couple hundred; a high-resolution page tiled into many crops can yield thousands. Two things make this hard for the language model.
First, the count is variable. The LLM would have to handle a wildly different image sequence length from one request to the next, which makes latency and memory unpredictable. Second, the count can be large, and attention cost grows with sequence length, so dumping thousands of patch tokens into the context is expensive on every layer.
These two pains compound. A request with a small thumbnail and a request with a tiled high-resolution scan would not just cost different amounts — they would have wildly different latency and memory footprints, making it hard to plan batching or guarantee a response time. The language model has no good way to absorb that swing on its own. Something upstream has to tame it.
The naive connector — an MLP that projects every patch one to one into the token stream — does exactly that. It is simple and preserves detail, but it hands the variability and the size straight to the LLM.
The resampler exists to stand in that gap. Its job is to turn an unpredictable, large input into a small, fixed one before the language backbone ever sees it. That single guarantee — a constant, known image budget — is what makes serving math tractable, because you can size context and latency against a number you chose rather than a number the user's image dictated.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- BLIP-2's Q-Former compressing a frozen vision encoder's features into 32 query tokens before the LLM.
- Flamingo's perceiver resampler mapping a variable number of visual features to a fixed set of latent tokens for cross-attention.
What an interviewer would ask next. Try answering before peeking at the approach.
QIn the cross-attention step, which side supplies the queries and which supplies the keys and values?
The fixed learnable tokens are the queries; the encoder's patch features are the keys and values, so the output length follows the query count, not the patch count.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Thinking the query tokens come from the image. They are learned parameters of the connector, fixed in count, and they read over the patch features rather than being produced by them.
60 second bullets to scan on the way to the call.
Why a variable patch count is a problem for the language model
What the learnable query tokens are and where they come from
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.