Compare projection adapter, cross-attention, and early fusion as ways to combine vision and language

Short answer·Hard·4.0 · 0·~3 min·Asked atCresta Mckinsey Tesla

Attempt it

A VLM has to merge vision encoder features with a text LLM. Compare projection adapter (LLaVA style), cross-attention (Flamingo style), and unified early fusion (Fuyu / native GPT-4o), and explain the tradeoff each one makes around sequence length, training cost, and capability.

Free · 2 AI evals / day

All three strategies connect a vision encoder to a text LLM, and the comparison is really a three-way tradeoff between sequence length, training cost, and how deeply the modalities mix. **Projection-adapter**, the LLaVA-style design, runs a small MLP that maps the encoder's outputs into the LLM's token space. The visual features become extra tokens in the sequence. It is cheap to train and the most common open design, but every image spends sequence length, so sending many images quickly fills the context window. Cross-attention, the Flamingo-style design, keeps the LLM's own text tokens and adds gated layers through which those tokens attend to the visual features. Because the features stay outside the sequence, image count no longer inflates context. The cost is extra parameters and a harder training process. Unified early fusion, used by Fuyu and native models like GPT-4o, tokenizes image patches into one stream from the start, with no separate encoder. It gives the tightest cross-modal reasoning and low latency, but it needs heavy joint training from scratch. The throughline: you are choosing which cost you can afford — context length, training effort, or the inability to reuse a pretrained vision tower.

These three strategies all merge encoder features with a text LLM, but each makes a different bet on the triangle of **sequence-length cost, training cost, and depth of modality mixing**. The projection-adapter (LLaVA-style) maps encoder outputs through a small MLP into the LLM's token embedding space, so visual features ride in the sequence as extra tokens. It is cheap to train — the vision tower and LLM can stay mostly frozen — and it dominates open models. Its weakness is that every image spends sequence length, so image count and resolution directly multiply context. A multi-image or tiled high-resolution request overflows the window. Cross-attention (Flamingo-style) keeps the LLM's text tokens and inserts gated cross-attention layers through which those tokens attend to visual features held outside the sequence. This decouples sequence length from image count, so many images or long interleaved documents stay tractable. The price is added parameters and a harder optimization, since the new layers must inject visual signal without destabilizing the language model. Unified early fusion (Fuyu, native GPT-4o and Gemini) tokenizes image patches into the same stream from the start with no separate encoder. It delivers the tightest cross-modal reasoning and low latency, but it needs heavy joint training from scratch and forfeits the ability to drop in a pretrained vision tower. The decision principle: pick the strategy whose cost you can pay for the workload. Single-image chat favors the cheap projection-adapter; many-image or document-heavy products favor cross-attention or a resampler; a from-scratch frontier model can afford early fusion for the best mixing.

TL;DR

Fusion trades off on one triangle: projection-adapter is cheap but spends context per image, cross-attention decouples image count at higher training cost, and early fusion mixes tightest but needs full joint training.

Memory aid

Easy to grasp

Imagine teaching a fluent storyteller to also use pictures. The quick way: turn each picture into a paragraph of words and slip it into the story — easy, but long pictures make the story balloon. The flexible way: leave the story untouched and let the storyteller peek at a gallery on the wall whenever needed — you can hang any number of pictures without lengthening the story, but you have to train that peeking habit carefully. The deepest way: raise the storyteller from birth to think in pictures and words as one thing — the most natural blend, but you cannot reuse a storyteller who already learned words only. Choosing a fusion strategy means picking which of those costs you can afford.

Key concepts

Multimodal AI Attention

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Fusion strategy is where a vision-language model's whole cost structure is decided, which is why staff-level interviews park on it. The wrong instinct is to ask which design is best. There is no best — there is a triangle of costs, and each strategy is a deliberate choice about which cost you can afford for a given product.

The three costs are context budget (how much sequence length visual content consumes), training budget (how much compute and data the connector demands), and mixing depth (how tightly the two modalities interact, which sets the capability ceiling). This deep dive treats the comparison as a constrained optimization over those three axes. It quantifies the projection-adapter's context cost with real token math, shows how cross-attention and early fusion buy context independence and mixing depth at a training-cost premium, and adds the resampler lever that lets you tune the projection path's token budget. The goal is to leave you able to recommend a strategy from a workload description, not recite three definitions.

The cost triangle that defines the choice

Start by naming the three axes, because every strategy is a point in this space and the comparison collapses without it.

Context budget. Visual content can consume sequence length. If it does, then image count and resolution eat into the window the model has for text and reasoning. This is a runtime cost, paid on every request.

Training budget. Connecting an encoder to an LLM can be as cheap as training a two-layer MLP or as expensive as jointly pretraining the entire model from scratch. This is a one-time cost, but it gates how quickly you can iterate and whether you can reuse pretrained components.

Mixing depth. How early and how often the modalities interact sets the capability ceiling. Features spliced in late mix less than patches that share every layer with text. Deeper mixing tends to mean stronger cross-modal reasoning.

The reason there is no universal winner is that these axes trade against each other. Cheap training tends to come with a context penalty. Context independence tends to cost training effort. The tightest mixing tends to cost the most training and the most modularity. A senior answer states the triangle first, then places each strategy on it, because the placement is what the design decision actually turns on.

Framing it as a triangle also guards against a common interview trap: declaring one strategy simply "best." The moment you name the three axes, it becomes obvious that a workload chooses the winner. A single-image consumer chatbot and a from-scratch frontier model are optimizing for different corners, and a recommendation that ignores which corner the product lives in is not a senior answer, however confidently it is delivered.

Projection-adapter and the arithmetic of context cost

Cross-attention and early fusion: paying training for context or mixing

The resampler lever and choosing by workload

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Axis	Projection-adapter (LLaVA)	Cross-attention (Flamingo)	Early fusion (Fuyu / GPT-4o)
Visual feature placement	Inside the token sequence	Outside, via inserted attention layers	One stream, no separate encoder
Sequence-length cost	Scales with image count and resolution	Decoupled from image count	Moderate, tightest mixing
Training cost	Low — MLP on frozen towers	Medium — added params, harder to train	High — joint training from scratch
Best fit	Single-image chat, cheap iteration	Many-image and interleaved inputs	Frontier native multimodal models

Real products, models, and research that use this idea.

LLaVA's projection-adapter trains a two-layer MLP on a frozen ViT and LLM, the cheapest path to a capable open VLM
Flamingo's gated cross-attention let one model ingest interleaved image-text few-shot prompts without inflating sequence length

What an interviewer would ask next. Try answering before peeking at the approach.

QWhere does a resampler like a Q-Former sit in this taxonomy, and what does it change?

It is a compression stage between the encoder and the connector, not a fourth fusion family. It maps the encoder's many patch features to a small fixed set of query tokens, capping tokens per image. Inside the projection path it blunts the context penalty; it can also feed cross-attention. Frame it as a lever on the token budget, orthogonal to where features enter.

1 more follow-up an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Recommending a projection-adapter for a many-image product because it is cheap to train, ignoring that each image multiplies the context length until requests overflow the window.

60 second bullets to scan on the way to the call.

Name the three axes the fusion strategies trade off against
Explain why projection-adapter context scales with image count and resolution

Primary sources. Browse if you want the original framing.