Match each vision-language fusion strategy to its defining property

Match pairs·Medium·4.0 · 0·~2 min·Asked atDeepseek Robust Intelligence Two Sigma

Attempt it

Drag each answer to line up with its matching prompt

Projection adapter (LLaVA style)

Splits the image into patches and produces the visual embeddings that the fusion strategy consumes

Cross-attention (Flamingo style)

Small MLP maps encoder outputs into the LLM token space; cheap to train but spends sequence length per image

Unified early fusion (Fuyu / native GPT-4o)

Image patches tokenized into one stream with no separate encoder; strongest cross-modal mixing, heaviest joint training

Vision encoder (ViT)

LLM text tokens attend to visual features through inserted layers, decoupling sequence length from image count

All three strategies answer one question: how do the visual features from the encoder reach the language model? The defining property of each is *where* the image information enters. **Projection-adapter**, the LLaVA-style design, runs a small MLP that maps encoder outputs into the LLM's token space. The image then sits in the sequence as extra tokens. It is cheap to train because the language model barely changes, but every image spends sequence length, so many images inflate the context window. Cross-attention, the Flamingo-style design, keeps the language model's own text tokens and lets them attend to visual features through inserted layers. The visual features live outside the sequence, so image count is decoupled from sequence length — you can feed many images without filling the context. Unified early fusion, used by Fuyu and native models like GPT-4o, tokenizes image patches into the same stream from the start, with no separate encoder. This gives the tightest cross-modal mixing but needs heavy joint training. The fourth item, the vision encoder, is not a fusion strategy at all — it is the ViT that splits the image into patches and produces the embeddings the fusion strategy consumes.

Matching these correctly turns on **where the visual features enter the language model**, because that single placement decision sets each strategy's cost profile. The **projection-adapter** (LLaVA-style) maps encoder outputs through a small MLP into the LLM token space, so visual features become extra tokens inside the sequence. Its defining property is the tradeoff: cheap to train and simple, but each image consumes sequence length, so image count multiplies context. This is why it dominates open models yet struggles with many-image requests. Cross-attention (Flamingo-style) keeps the LLM's text tokens and inserts gated layers through which those tokens attend to visual features held outside the sequence. Its defining property is decoupling — sequence length no longer scales with image count, so you can feed many images, at the price of added parameters and harder training. Unified early fusion (Fuyu, native GPT-4o and Gemini) tokenizes image patches directly into one stream with no separate encoder. Its defining property is the tightest cross-modal mixing and low latency, paid for with heavy joint training from scratch. The fourth entry, the **vision encoder** (a ViT), is deliberately not a fusion strategy. It splits the image into patches and produces the embeddings that whichever fusion strategy then consumes. Keeping it distinct is the point of the question — the encoder produces features; the fusion strategy decides how they meet the language.

TL;DR

Projection-adapter puts image features in the token sequence (cheap, costs context per image), cross-attention reaches them from outside (decouples image count), and early fusion uses one patch stream (tightest mixing).

Memory aid

Easy to grasp

Imagine three ways to add pictures to a story you are telling out loud. The first way is to read the picture's description out loud as extra words in the story — easy to set up, but a long photo eats a lot of speaking time. The second way is to keep the story exactly as it is and just glance at the pictures on the wall whenever you need them — you can hang as many pictures as you like without making the story longer. The third way is to never separate pictures from words at all; you learn to speak in a language where pictures and words are the same kind of thing from the start — the most powerful, but you have to relearn the whole language. Those three are the fusion strategies.

Key concepts

Multimodal AI Transformer Architecture

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Fusion strategy is the part of a vision-language model that interviewers probe hardest, because it is where capability, cost, and training difficulty are all decided at once. The matching version of the question is a trap if you treat it as vocabulary recall — the four rows look similar enough that a candidate who memorized names without the underlying mechanism will swap two of them.

The key that unlocks the whole question is one idea: each strategy is defined by where the visual features enter the language model, and everything else — context cost, training cost, how deeply the modalities mix — follows from that placement. This deep dive walks through the three fusion families along that single axis, then separates the one distractor, the vision encoder, which is not a fusion strategy at all but the component that feeds them.

The organizing axis: where visual features enter

Every fusion strategy has to answer the same question. The vision encoder has produced a set of visual embeddings; how do those embeddings reach the language model so it can reason over them alongside text?

There are exactly three answers, and they line up with three positions relative to the token sequence the language model processes. The first answer is inside the sequence: convert the visual embeddings into things that look like tokens and splice them in. The second is outside the sequence: leave the text tokens alone and let them reach the visual features through a separate attention pathway. The third is no separation at all: never have a distinct visual embedding to place, because image patches and text are the same kind of token from the very first layer.

This axis is what makes the matching tractable. Once you locate a strategy on it, the cost profile is forced. Inside the sequence means image content competes for context length. Outside the sequence means context length is independent of how many images you send. No separation means the modalities interact at every layer, which is powerful but means the whole model must be trained jointly. Memorize the axis, not the rows.

Projection-adapter: cheap, but it eats the context

Cross-attention and early fusion: the other two answers

The distractor: why the ViT is not a fusion strategy

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Strategy	Where visuals enter	Sequence-length cost	Training cost
Projection-adapter (LLaVA)	Inside the token sequence as extra tokens	High — scales with image count	Low — LLM barely changes
Cross-attention (Flamingo)	Outside, reached via inserted gated layers	Decoupled from image count	Medium — added params, harder to train
Unified early fusion (Fuyu / GPT-4o)	Patches tokenized into one stream, no encoder	Moderate, but tightest mixing	High — heavy joint training from scratch

Real products, models, and research that use this idea.

LLaVA popularized the projection-adapter as a two-layer MLP connector, training cheaply on a frozen vision tower and LLM
Flamingo introduced gated cross-attention layers, letting one model ingest many interleaved images without inflating sequence length

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy did open-source VLMs converge on the projection-adapter despite its context cost?

Trace the economics: a projection MLP trains in hours on a frozen pretrained ViT plus a frozen or lightly tuned LLM, needing only modest visual instruction data. Cross-attention and early fusion need far more compute and data. For single-image chat, the context cost rarely bites, so the cheap option wins on cost per capability.

1 more follow-up an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Assuming projection-adapter and cross-attention cost the same in context length, when only the projection design spends sequence tokens for every image.

60 second bullets to scan on the way to the call.

State where each fusion strategy injects visual features relative to the token sequence
Explain why projection-adapter spends sequence length per image

Primary sources. Browse if you want the original framing.