What does the projection/connector layer do in a vision-language model, and why do engineers call it the key design choice rather than just plumbing?
Free · 2 AI evals / day
A vision-language model has a vision encoder that turns an image into feature vectors and an LLM that works in its own token embedding space. The **connector** is the layer that maps the vision features into the LLM's space so the model can attend to image features alongside text.
It is called the key design choice because it decides three things at once. First, capability: how much visual information actually reaches the LLM. Second, cost: how many vision tokens each image becomes, which sets context length and latency. Third, training difficulty: how much of the encoder and LLM you have to retrain.
The simplest connector is an MLP that projects every image patch into a token. That is cheap to build but spends sequence length on every patch. Other designs, like a resampler, compress the image into a fixed small number of tokens. The choice is a real tradeoff between how rich and how expensive the image is, which is why it is not just plumbing.
The **connector** maps the vision encoder's output features into the LLM's token embedding space, so visual features sit in the sequence and the model attends over them alongside text tokens. Mechanically it is a translation layer, but its design ripples into the rest of the system.
That is why engineers treat it as the pivotal choice rather than plumbing. It sets capability — how richly visual detail reaches the LLM. It sets cost — how many vision tokens you spend per image, which directly drives context length, latency, and price. And it sets training difficulty — how much of the encoder and LLM must be unfrozen and retrained.
The design space makes the tradeoff concrete. A simple MLP projection, the LLaVA-style choice, forwards one token per patch: cheap to train but it burns sequence length, and high-resolution images explode the token count. A resampler or Q-Former compresses many patches into a fixed small set of query tokens, capping cost at the risk of dropping detail. Cross-attention, the Flamingo-style choice, keeps visual features out of the sequence and lets the LLM attend to them through inserted layers, decoupling image count from context length. Each picks a different point on the capability versus cost curve.
Quick reference
TL;DR
The connector maps vision features into the LLM's token space. It is pivotal because its design sets capability, per-image token cost, and how much retraining you need — all at once.
Imagine a chef (the language model) who only reads recipes written in their own handwriting. A photographer hands over pictures, but the chef can't read pictures. The connector is the assistant who rewrites what's in each photo into the chef's handwriting so the chef can cook with it.
The twist: how that assistant works decides a lot. Copy every tiny detail and the chef drowns in pages to read. Summarize too hard and the chef misses things. So the assistant's style — verbose or terse — quietly sets how much work the chef does and how good the dish is.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
A fixed-budget resampler when the task is dense OCR or chart reading Compressing to 32-256 tokens drops the fine spatial detail those tasks need; patch-forwarding preserves more.
MLP patch-forwarding for many high-resolution images per request Token count explodes with resolution and image count, blowing the context budget and latency; a resampler caps it.
Cross-attention when you want to keep the base LLM untouched It inserts new gated layers into the LLM, so it is more invasive than a thin external projection you can train alone.
Quick scan · 2–4 min
Everything important, quickly.
About 6 minutes: define what the connector maps and why, then walk the three budgets it sets, and contrast MLP, resampler, and cross-attention with their token-cost consequences.
Side-by-side
Connector
Vision tokens per image
Tradeoff
MLP projection (LLaVA-style)
One per patch — grows with resolution
Cheap to train, detail-preserving, but burns context length
Resampler / Q-Former
Fixed small set (e.g. 32-256)
Caps cost at any resolution, risks losing fine detail, harder to train
Cross-attention (Flamingo-style)
Zero added to the sequence
Decouples image count from context length, more invasive to the LLM
Real-world examples
Real products, models, and research that use this idea.
1LLaVA — uses a simple MLP projection from a CLIP encoder into the LLM token space.
2Flamingo — uses gated cross-attention layers so image count does not grow sequence length.
3Qwen3-VL and Pixtral — pair patch projection with dynamic resolution tiling to manage per-image token cost.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QHow does the connector choice interact with high-resolution tiling like AnyRes?
Sign in or go Pro to see the approach for this follow-up.
QWhy can a thin MLP connector be trained while the encoder and LLM stay frozen?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Calling the connector "just a projection" or plumbing. Its design directly sets how many vision tokens an image costs, which drives context length, latency, and price.