Why is the connector the pivotal design choice in a vision-language model?
What does the projection/connector layer do in a vision-language model, and why do engineers call it the key design choice rather than just plumbing?
The connector maps vision features into the LLM's token space. It is pivotal because its design sets capability, per-image token cost, and how much retraining you need — all at once.
Imagine a chef (the language model) who only reads recipes written in their own handwriting. A photographer hands over pictures, but the chef can't read pictures. The connector is the assistant who rewrites what's in each photo into the chef's handwriting so the chef can cook with it. The twist: how that assistant works decides a lot. Copy every tiny detail and the chef drowns in pages to read. Summarize too hard and the chef misses things. So the assistant's style — verbose or terse — quietly sets how much work the chef does and how good the dish is.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
A vision-language model is often described as "a vision encoder bolted onto an LLM," which makes the connector sound like a wire between two boxes. That framing hides the most consequential decision in the whole design. The encoder and the LLM are usually pretrained and largely frozen; the connector is where the actual engineering choices live.
The reason this question shows up in senior interviews is that the connector is the one module that trades capability against cost against trainability, and those three rarely move in the same direction. A choice that gives the LLM richer visual detail usually costs more tokens; a choice that caps tokens usually risks losing detail; a choice that minimizes retraining usually limits how deeply the modalities can mix. Understanding the connector means understanding the entire cost structure of serving a VLM, because the per-image token count it produces propagates into context length, KV-cache size, prefill latency, and the dollar price of every request.
This deep dive covers what the connector does mechanically, why its design sets per-image token cost and therefore latency and price, how the three main connector families differ, and how it interacts with high-resolution handling — the place where token budgets actually blow up in production document and chart workloads.
What the connector actually does
The vision encoder, typically a Vision Transformer, splits an image into patches and outputs a sequence of feature vectors — one per patch. Those vectors live in the encoder's own space, with the encoder's own dimension and its own learned meaning. The LLM, meanwhile, operates on token embeddings of a different dimension and a different learned geometry.
The connector bridges the two. At minimum it projects each vision feature into the LLM's token embedding dimension. Once projected, those vectors are inserted into the input sequence and treated like extra tokens, so the LLM's attention can mix image and text freely.
That "treated like tokens" part is the mechanism that makes a VLM work. The language backbone never gets a special image pathway; it just sees a longer sequence where some positions came from pixels. So the connector's real job is to produce vectors that sit convincingly in token space — close enough to real token embeddings that the frozen LLM can reason over them without being retrained from scratch.
This framing also explains why the connector is trainable on a small budget. Both the encoder and the LLM arrive pretrained. The connector is the only piece that must learn the mapping between their two spaces, so a typical recipe freezes both giants and trains just the connector on image-text alignment data, then optionally unfreezes the LLM for a light instruction-tuning pass. The connector is small, but it is the seam the whole system is stitched at.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
| Connector | Vision tokens per image | Tradeoff |
|---|---|---|
| MLP projection (LLaVA-style) | One per patch — grows with resolution | Cheap to train, detail-preserving, but burns context length |
| Resampler / Q-Former | Fixed small set (e.g. 32-256) | Caps cost at any resolution, risks losing fine detail, harder to train |
| Cross-attention (Flamingo-style) | Zero added to the sequence | Decouples image count from context length, more invasive to the LLM |
Real products, models, and research that use this idea.
- LLaVA — uses a simple MLP projection from a CLIP encoder into the LLM token space.
- Flamingo — uses gated cross-attention layers so image count does not grow sequence length.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow does the connector choice interact with high-resolution tiling like AnyRes?
Trace how more tiles multiply patch tokens under an MLP, and how a resampler caps that, framing the resolution versus token budget tradeoff.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Calling the connector "just a projection" or plumbing. Its design directly sets how many vision tokens an image costs, which drives context length, latency, and price.
60 second bullets to scan on the way to the call.
What space the connector maps vision features into and why
How visual features end up attended alongside text tokens
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.