1Split the input image into a grid of fixed-size patches
2Concatenate the projected visual tokens with the text tokens and let the LLM attend over both
3Run the patch embeddings through the vision transformer encoder
4Linearly embed each patch into a vector and add positional information
5Project the encoder outputs into the LLM's token embedding space via the connector
The pipeline has five stages, and each one feeds the next.
First, the image is cut into a grid of fixed-size **patches**, usually 14x14 or 16x16 pixels each. A 336x336 image at 14-pixel patches becomes a 24x24 grid, so 576 patches.
Second, every patch is flattened and run through a small linear layer into a vector, and positional information is added so the model knows where each patch sat in the grid.
Third, those patch vectors go through the vision transformer encoder. The encoder lets patches attend to each other, so each output vector now carries context from the whole image, not just its own square.
Fourth, a connector (a small projection or MLP) maps those encoder outputs into the LLM's token embedding space, so a visual vector now has the same width and meaning as a text token.
Fifth, the projected visual tokens are placed alongside the text tokens, and the language model attends over both together. The LLM never sees raw pixels; it only sees these projected vectors as extra positions in its sequence.
The correct order is **patchify, embed, encode, project, then attend** — and the dependency chain is what locks it.
You start by splitting the image into a grid of fixed-size patches (say 14x14 pixels). Each patch is linearly embedded into a vector, and positional information is added so the grid layout survives. Those embeddings then pass through the vision transformer, where self-attention mixes information across patches so each output vector is context-aware.
The key bridge is the connector. The encoder emits vectors in its own hidden dimension, which rarely matches the LLM's embedding width. A projection layer (LLaVA uses a small MLP) maps the encoder outputs into the LLM's token embedding space. Only then can the visual vectors sit next to text tokens.
The last stage concatenates these projected visual tokens with the text tokens, and the LLM attends over the combined sequence. Each step strictly depends on the previous output: you cannot encode patches you have not cut, and you cannot project features the encoder has not produced. The whole point is that the language model treats the image as extra tokens, never as pixels.
Quick reference
TL;DR
Patchify the image, embed and position the patches, encode them with a ViT, project the features into the LLM's token space, then let the LLM attend over image and text tokens together.
Imagine handing someone a giant photo, but they can only read index cards. First you cut the photo into a neat grid of little squares. You write a short description on a card for each square, then a vision expert studies all the cards together and rewrites them so they make sense as a set. Finally you translate those cards into the exact handwriting the reader already understands and shuffle them into the reader's existing stack of word cards. Now the reader flips through picture cards and word cards in one pile, treating them the same way. The order matters: you cannot study cards before cutting the photo, and you cannot translate cards the expert has not written yet.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Unified early-fusion models like Fuyu that tokenize patches straight into the LLM stream There is no separate vision encoder, so the distinct encode then project stages collapse; patches enter the same attention stack as text from the start.
Cross-attention connector designs (Flamingo-style) where the LLM keeps its own token count Visual features are not concatenated as extra positions; the LLM attends to them through inserted gated cross-attention layers, changing the final fusion step.
Image generation rather than understanding Producing pixels needs a diffusion or autoregressive image decoder; this encode and project path only covers reading an image, not making one.
Quick scan · 2–4 min
Everything important, quickly.
Name the five stages in order, justify each dependency by tracking the data shape, and explain why the connector sits between the encoder and the joint-attention step rather than anywhere else.
Real-world examples
Real products, models, and research that use this idea.
1LLaVA uses a CLIP-pretrained ViT plus a small MLP connector, the canonical open-source version of exactly this five-stage path.
2GPT-5.5 and Gemini 3.1 Pro are natively multimodal but still patchify and encode images before the language stack attends over the resulting tokens.
3Qwen3-VL and other modern open VLMs follow the same patchify, encode, project order, with dynamic-resolution tiling added on top of the patch grid.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy is a positional embedding added to the patches before the encoder if the ViT already uses self-attention?
Sign in or go Pro to see the approach for this follow-up.
QWhat changes in this ordering when the model uses a cross-attention connector instead of a projection MLP?
Sign in or go Pro to see the approach for this follow-up.
QHow does AnyRes or dynamic-resolution tiling interact with the patchify step?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Putting the connector projection before the vision encoder. The connector maps encoder outputs, so the ViT must run first or there is nothing to project.