All three strategies answer one question: how do the visual features from the encoder reach the language model? The defining property of each is *where* the image information enters.
**Projection-adapter**, the LLaVA-style design, runs a small MLP that maps encoder outputs into the LLM's token space. The image then sits in the sequence as extra tokens. It is cheap to train because the language model barely changes, but every image spends sequence length, so many images inflate the context window.
Cross-attention, the Flamingo-style design, keeps the language model's own text tokens and lets them attend to visual features through inserted layers. The visual features live outside the sequence, so image count is decoupled from sequence length — you can feed many images without filling the context.
Unified early fusion, used by Fuyu and native models like GPT-4o, tokenizes image patches into the same stream from the start, with no separate encoder. This gives the tightest cross-modal mixing but needs heavy joint training. The fourth item, the vision encoder, is not a fusion strategy at all — it is the ViT that splits the image into patches and produces the embeddings the fusion strategy consumes.
Matching these correctly turns on **where the visual features enter the language model**, because that single placement decision sets each strategy's cost profile.
The **projection-adapter** (LLaVA-style) maps encoder outputs through a small MLP into the LLM token space, so visual features become extra tokens inside the sequence. Its defining property is the tradeoff: cheap to train and simple, but each image consumes sequence length, so image count multiplies context. This is why it dominates open models yet struggles with many-image requests.
Cross-attention (Flamingo-style) keeps the LLM's text tokens and inserts gated layers through which those tokens attend to visual features held outside the sequence. Its defining property is decoupling — sequence length no longer scales with image count, so you can feed many images, at the price of added parameters and harder training.
Unified early fusion (Fuyu, native GPT-4o and Gemini) tokenizes image patches directly into one stream with no separate encoder. Its defining property is the tightest cross-modal mixing and low latency, paid for with heavy joint training from scratch.
The fourth entry, the **vision encoder** (a ViT), is deliberately not a fusion strategy. It splits the image into patches and produces the embeddings that whichever fusion strategy then consumes. Keeping it distinct is the point of the question — the encoder produces features; the fusion strategy decides how they meet the language.
Quick reference
TL;DR
Projection-adapter puts image features in the token sequence (cheap, costs context per image), cross-attention reaches them from outside (decouples image count), and early fusion uses one patch stream (tightest mixing).
Imagine three ways to add pictures to a story you are telling out loud. The first way is to read the picture's description out loud as extra words in the story — easy to set up, but a long photo eats a lot of speaking time. The second way is to keep the story exactly as it is and just glance at the pictures on the wall whenever you need them — you can hang as many pictures as you like without making the story longer. The third way is to never separate pictures from words at all; you learn to speak in a language where pictures and words are the same kind of thing from the start — the most powerful, but you have to relearn the whole language. Those three are the fusion strategies.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Projection-adapter for requests with many high-resolution images Each image spends sequence tokens, so the context window overflows; cross-attention or a resampler fits the workload better.
Cross-attention when you only ever send a single image and want minimal training The added parameters and harder optimization buy decoupling you never use; a projection MLP is cheaper and sufficient.
Early fusion when you must reuse a strong pretrained vision tower Early fusion has no separate encoder, so you cannot drop in an existing ViT; you would retrain perception from scratch.
Quick scan · 2–4 min
Everything important, quickly.
Read each pair as a statement about where visual features enter the language model. Projection-adapter puts them inside the sequence (cheap, costs context). Cross-attention reaches them from outside (decouples image count). Early fusion tokenizes patches into one stream with no encoder (tightest mixing, heaviest training). Then separate the ViT, which is upstream perception, not a fusion strategy at all.
Side-by-side
Strategy
Where visuals enter
Sequence-length cost
Training cost
Projection-adapter (LLaVA)
Inside the token sequence as extra tokens
High — scales with image count
Low — LLM barely changes
Cross-attention (Flamingo)
Outside, reached via inserted gated layers
Decoupled from image count
Medium — added params, harder to train
Unified early fusion (Fuyu / GPT-4o)
Patches tokenized into one stream, no encoder
Moderate, but tightest mixing
High — heavy joint training from scratch
Real-world examples
Real products, models, and research that use this idea.
1LLaVA popularized the projection-adapter as a two-layer MLP connector, training cheaply on a frozen vision tower and LLM
2Flamingo introduced gated cross-attention layers, letting one model ingest many interleaved images without inflating sequence length
3Fuyu removed the separate vision encoder and fed image patches directly into the decoder as an early-fusion design
4GPT-5.5 and Gemini 3.1 Pro are described as natively multimodal, mixing modalities early rather than bolting an encoder onto a text LLM
5Open VLMs in 2026 still lean on projection-adapter connectors because they are cheap to train on a pretrained ViT plus LLM
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy did open-source VLMs converge on the projection-adapter despite its context cost?
Sign in or go Pro to see the approach for this follow-up.
QWhat changes about the connector choice when the product must handle many images per request?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Assuming projection-adapter and cross-attention cost the same in context length, when only the projection design spends sequence tokens for every image.