- 1Split the input image into a grid of fixed size patches
- 2Concatenate the projected visual tokens with the text tokens and let the LLM attend over both
- 3Run the patch embeddings through the vision transformer encoder
- 4Linearly embed each patch into a vector and add positional information
- 5Project the encoder outputs into the LLM's token embedding space via the connector
Patchify the image, embed and position the patches, encode them with a ViT, project the features into the LLM's token space, then let the LLM attend over image and text tokens together.
Imagine handing someone a giant photo, but they can only read index cards. First you cut the photo into a neat grid of little squares. You write a short description on a card for each square, then a vision expert studies all the cards together and rewrites them so they make sense as a set. Finally you translate those cards into the exact handwriting the reader already understands and shuffle them into the reader's existing stack of word cards. Now the reader flips through picture cards and word cards in one pile, treating them the same way. The order matters: you cannot study cards before cutting the photo, and you cannot translate cards the expert has not written yet.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Interviewers love this ordering question because it forces you to reason about a pipeline rather than recite a definition. A vision-language model is two networks stitched together, and the seam between them is where most candidates get vague. Getting the order right means you understand what each stage produces and consumes.
The deeper reason the order is fixed is that every stage transforms the data shape, and the next stage can only run on the shape the previous one emitted. Pixels become patch vectors, patch vectors become context-aware features, features become LLM-space tokens, and tokens get attended over. Reverse any arrow and the shapes no longer line up.
This walkthrough builds the pipeline one stage at a time, explains why the connector is the architecturally interesting piece, and connects the canonical projection design to the cross-attention and early-fusion alternatives you should be ready to name.
From pixels to a grid of patch vectors
A transformer consumes a sequence of vectors, but an image is a 2D grid of pixels. The first job is to turn one into the other.
The image is split into a regular grid of fixed-size patches. A common choice is 14x14 pixels per patch on a 336x336 input, which gives a 24x24 grid, so 576 patches. Each patch is flattened and pushed through a single linear layer, the patch embedding, producing one vector per patch in the encoder's hidden dimension.
The position problem. Self-attention treats its inputs as an unordered set, so two patches swapped in the grid would produce the same output. That would be a disaster for an image, where the top-left and bottom-right squares mean different things. A positional embedding is added to each patch vector to encode where it sat in the grid.
At the end of this stage you have a sequence of patch vectors with position baked in. No attention has happened yet; each vector still only knows about its own square. That context-mixing is the next stage's job, which is exactly why patchify and embed must come before the encoder runs.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- LLaVA uses a CLIP-pretrained ViT plus a small MLP connector, the canonical open-source version of exactly this five-stage path.
- GPT-5.5 and Gemini 3.1 Pro are natively multimodal but still patchify and encode images before the language stack attends over the resulting tokens.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy is a positional embedding added to the patches before the encoder if the ViT already uses self-attention?
Self-attention is permutation-invariant, so without positions the encoder cannot tell a top-left patch from a bottom-right one. The positional embedding restores the 2D grid the layout depends on.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Putting the connector projection before the vision encoder. The connector maps encoder outputs, so the ViT must run first or there is nothing to project.
60 second bullets to scan on the way to the call.
Can you list the five stages in order and say what shape the data has after each one?
Why must the vision encoder run before the connector projection?
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.