A VLM has to merge vision-encoder features with a text LLM. Compare projection-adapter (LLaVA-style), cross-attention (Flamingo-style), and unified early fusion (Fuyu / native GPT-4o), and explain the tradeoff each one makes around sequence length, training cost, and capability.
Free · 2 AI evals / day
All three strategies connect a vision encoder to a text LLM, and the comparison is really a three-way tradeoff between sequence length, training cost, and how deeply the modalities mix.
**Projection-adapter**, the LLaVA-style design, runs a small MLP that maps the encoder's outputs into the LLM's token space. The visual features become extra tokens in the sequence. It is cheap to train and the most common open design, but every image spends sequence length, so sending many images quickly fills the context window.
Cross-attention, the Flamingo-style design, keeps the LLM's own text tokens and adds gated layers through which those tokens attend to the visual features. Because the features stay outside the sequence, image count no longer inflates context. The cost is extra parameters and a harder training process.
Unified early fusion, used by Fuyu and native models like GPT-4o, tokenizes image patches into one stream from the start, with no separate encoder. It gives the tightest cross-modal reasoning and low latency, but it needs heavy joint training from scratch. The throughline: you are choosing which cost you can afford — context length, training effort, or the inability to reuse a pretrained vision tower.
These three strategies all merge encoder features with a text LLM, but each makes a different bet on the triangle of **sequence-length cost, training cost, and depth of modality mixing**.
The projection-adapter (LLaVA-style) maps encoder outputs through a small MLP into the LLM's token embedding space, so visual features ride in the sequence as extra tokens. It is cheap to train — the vision tower and LLM can stay mostly frozen — and it dominates open models. Its weakness is that every image spends sequence length, so image count and resolution directly multiply context. A multi-image or tiled high-resolution request overflows the window.
Cross-attention (Flamingo-style) keeps the LLM's text tokens and inserts gated cross-attention layers through which those tokens attend to visual features held outside the sequence. This decouples sequence length from image count, so many images or long interleaved documents stay tractable. The price is added parameters and a harder optimization, since the new layers must inject visual signal without destabilizing the language model.
Unified early fusion (Fuyu, native GPT-4o and Gemini) tokenizes image patches into the same stream from the start with no separate encoder. It delivers the tightest cross-modal reasoning and low latency, but it needs heavy joint training from scratch and forfeits the ability to drop in a pretrained vision tower.
The decision principle: pick the strategy whose cost you can pay for the workload. Single-image chat favors the cheap projection-adapter; many-image or document-heavy products favor cross-attention or a resampler; a from-scratch frontier model can afford early fusion for the best mixing.
Quick reference
TL;DR
Fusion trades off on one triangle: projection-adapter is cheap but spends context per image, cross-attention decouples image count at higher training cost, and early fusion mixes tightest but needs full joint training.
Imagine teaching a fluent storyteller to also use pictures. The quick way: turn each picture into a paragraph of words and slip it into the story — easy, but long pictures make the story balloon. The flexible way: leave the story untouched and let the storyteller peek at a gallery on the wall whenever needed — you can hang any number of pictures without lengthening the story, but you have to train that peeking habit carefully. The deepest way: raise the storyteller from birth to think in pictures and words as one thing — the most natural blend, but you cannot reuse a storyteller who already learned words only. Choosing a fusion strategy means picking which of those costs you can afford.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Projection-adapter for a product that ingests dozens of high-resolution images per request Tokens per image times image count overflows the context window; the cheap training is no help once requests do not fit.
Cross-attention for a single-image chatbot built on a fixed budget The decoupling buys headroom you never use while costing added parameters and a harder, more expensive training run.
Early fusion when you must reuse and upgrade a pretrained vision encoder over time Early fusion has no separate encoder, so the modular swap-in path is gone; every perception upgrade means retraining the whole model.
Quick scan · 2–4 min
Everything important, quickly.
Set up the comparison as one triangle: sequence-length cost, training cost, and depth of modality mixing. Walk projection-adapter (cheap training, context scales with images), cross-attention (decoupled context, harder training), and early fusion (tightest mixing, heaviest training, no reusable encoder). Quantify the projection context cost as image count times tokens per image, add the resampler lever, then map each strategy to the workload it fits.
Side-by-side
Axis
Projection-adapter (LLaVA)
Cross-attention (Flamingo)
Early fusion (Fuyu / GPT-4o)
Visual feature placement
Inside the token sequence
Outside, via inserted attention layers
One stream, no separate encoder
Sequence-length cost
Scales with image count and resolution
Decoupled from image count
Moderate, tightest mixing
Training cost
Low — MLP on frozen towers
Medium — added params, harder to train
High — joint training from scratch
Best fit
Single-image chat, cheap iteration
Many-image and interleaved inputs
Frontier native multimodal models
Real-world examples
Real products, models, and research that use this idea.
1LLaVA's projection-adapter trains a two-layer MLP on a frozen ViT and LLM, the cheapest path to a capable open VLM
2Flamingo's gated cross-attention let one model ingest interleaved image-text few-shot prompts without inflating sequence length
3BLIP-2's Q-Former is a resampler that compresses image features to a small fixed query-token set before they enter the LLM
4Fuyu drops the separate encoder and feeds image patches straight into the decoder as a clean early-fusion design
5GPT-5.5 and Gemini 3.1 Pro are described as natively multimodal, mixing modalities early for tight cross-modal reasoning and low latency
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QWhere does a resampler like a Q-Former sit in this taxonomy, and what does it change?
Sign in or go Pro to see the approach for this follow-up.
QHow does high-resolution tiling (AnyRes) interact with the projection-adapter's context cost?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Recommending a projection-adapter for a many-image product because it is cheap to train, ignoring that each image multiplies the context length until requests overflow the window.