Why is the connector the pivotal design choice in a vision-language model?
What does the projection/connector layer do in a vision-language model, and why do engineers call it the key design choice rather than just plumbing?
The connector maps vision features into the LLM's token space. It is pivotal because its design sets capability, per-image token cost, and how much retraining you need, all at once.
Imagine a chef (the language model) who only reads recipes written in their own handwriting. A photographer hands over pictures, but the chef can't read pictures. The connector is the assistant who rewrites what's in each photo into the chef's handwriting so the chef can cook with it. The twist: how that assistant works decides a lot. Copy every tiny detail and the chef drowns in pages to read. Summarize too hard and the chef misses things. So the assistant's style (verbose or terse) sets how much work the chef does and how good the dish is.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
A vision-language model is often described as "a vision encoder bolted onto an LLM." That makes the connector sound like a wire between two boxes. That framing hides the most consequential decision in the whole design. The encoder and the LLM are usually pretrained and largely frozen. The main engineering choices are in the connector.
The connector is the one module that trades capability against cost against trainability, and those three rarely move in the same direction. A choice that gives the LLM richer visual detail usually costs more tokens. A choice that caps tokens usually risks losing detail. A choice that minimizes retraining usually limits how deeply the modalities can mix. Understanding the connector means understanding the entire cost structure of serving a VLM. The per-image token count it produces propagates into context length, KV-cache size, prefill latency, and the dollar price of every request.
What the connector does
The vision encoder, typically a Vision Transformer, splits an image into patches and outputs a sequence of feature vectors, one per patch. Those vectors live in the encoder's own space, with the encoder's own dimension and its own learned meaning. The LLM, meanwhile, operates on token embeddings of a different dimension and a different learned geometry.
The connector bridges the two. At minimum it projects each vision feature into the LLM's token embedding dimension. Once projected, those vectors are inserted into the input sequence and treated like extra tokens. The LLM's attention can then mix image and text freely.
That "treated like tokens" part is the mechanism that makes a VLM work. The language backbone never gets a special image pathway. It only sees a longer sequence where some positions came from pixels. So the connector's main job is to produce vectors that fit well in token space. They must be close enough to real token embeddings that the frozen LLM can reason over them without being retrained from scratch.
This framing also explains why the connector is trainable on a small budget. Both the encoder and the LLM arrive pretrained. The connector is the only piece that must learn the mapping between their two spaces. So a typical recipe freezes both large models and trains only the connector on image-text alignment data. It then optionally unfreezes the LLM for a light instruction-tuning pass. The connector is small, but it is the point where the whole system joins together.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
| Connector | Vision tokens per image | Tradeoff |
|---|---|---|
| MLP projection (LLaVA-style) | One per patch, grows with resolution | Cheap to train, detail-preserving, but uses up context length |
| Resampler / Q-Former | Fixed small set (e.g. 32-256) | Caps cost at any resolution, risks losing fine detail, harder to train |
| Cross-attention (Flamingo-style) | Zero added to the sequence | Decouples image count from context length, more invasive to the LLM |
Real products, models, and research that use this idea.
- LLaVA: uses a simple MLP projection from a CLIP encoder into the LLM token space.
- Flamingo: uses gated cross-attention layers so image count does not grow sequence length.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow does the connector choice interact with high-resolution tiling like AnyRes?
Trace how more tiles multiply patch tokens under an MLP, and how a resampler caps that, framing the resolution versus token budget tradeoff.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Calling the connector "just a projection" or plumbing. Its design directly sets how many vision tokens an image costs, which drives context length, latency, and price.
60 second bullets to scan on the way to the call.
What space the connector maps vision features into and why
How visual features end up attended alongside text tokens
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.